Not losing our RAG: a thoughtful approach to knowledge bots

For better or worse, Retrieval Augmented Generation (RAG) has become one of the defining use cases for early Gen AI. And why not? The idea of a chatbot that offers a fluent interface to what might be an otherwise obtuse and obscure knowledge base of thousands of documents is very appealing - all the advantages of a LLM without it making things up. However, there are enough cautionary tales of bots going rogue, giving bad or illegal advice to suggest it's not that simple.

In this talk, I will tell the story of the Policy Bot at the City of Cape Town, our very own RAG effort. How it started life as an inadvertent demo of hallucination, our first experiments with retrieval workflows, and finally, the implementation of something that we're reasonably proud of: an attractive WUI with a knowledge base of thousands of documents that tends to get the answer right. More importantly, we think that we have built a tool that helps our colleagues better serve the residents of Cape Town.

But don't worry, this isn't going to just be an elaborate humble-brag, there will also be some good technical details - how we came to understand the underlying conceptual problems of search and synthesis, our multi-layered content discovery approach, our evals framework, as well as all the great open source Python libraries and frameworks that we have used along the way.

This talk is for everyone that has built a RAG bot, is going to build a RAG bot, or is just interested in how we have tried to build responsibly with Generative AI.