TLTD #16 - The first 90 days
Building an answer engine
It’s been quite some time since my last post. To my existing subscribers — I really appreciate you hanging around. After some time off I plan to re-start my writing cadence and also launch a new podcast — more on this soon!
A number of things got in the way of my writing habit, most notably buying a new apartment and changing jobs! That’s kept me busy for several months!
The new job? I joined Optus, a telecommunications company in Australia, as Senior Director of AI Engineering. It’s a bit of a mouthful but basically I lead engineering in the AI division. I’ve been having so much fun since joining back in July that I thought I’d share a bit about my experience so far.
Timing is everything
As soon as I joined Optus I heard we were planning a Hackathon specifically targeted at leveraging AI. I got excited — hackathons are usually so much fun and this presented me with a great opportunity to get to know people, go deep into an interesting problem and create something together while drastically reducing the time it took me to onboard.
Starting a new job is never easy. One of the things I have advocated for over the years is that engineering leaders need to be technical. They don’t need to write code every day but need to be able to hold their own in a technical discussion and, most importantly, be able to drive a highly capable team towards a sensible technical outcome.
The AI Hackathon — which we called Hack[AI]thon — was my chance to show off a bit. My team and I took on a project that focused on improving how every employee at Optus finds information. We set ourselves the goal of building an internal version of Perplexity.
If you are not familiar, Perplexity’s CEO describes it as an answer engine. It’s a fairly different paradigm than using Google where you’re presented with several links to go through to try and find the information you need. Perplexity instead uses Large Language Models (LLMs) to answer your question directly, providing sources in the form of annotations and references.
Google has started providing search summaries, clearly moving into Perplexity’s territory. Currently, I still think Perplexity has an edge
Why rebuild Perplexity? Data sovereignty and privacy. We can’t risk our internal documents and information getting into the wrong hands, or worse, be used to train these LLMs. That’s why we wanted to create an internal tool that would make it safe for employees to dump any content and ask any question without worrying about data privacy.
We ended up finishing 2nd in the hackathon, out of 50 different teams, all with incredible ideas! Because of our placement we got the chance to present our work to Optus’ executive team which was a pretty great experience. I wasn’t really expecting to be exposed to that crowd so early in my tenure.
The solution
Alright so how did we actually build this thing? It comes down to three key technologies:
Large Language Models (LLMs) like ChatGPT for creating conversational agents that could intelligently answer questions. We ended up using LLaMa 3.2 8b.
Model serving software like Ollama, which allowed us to run LLMs locally. This was particularly appealing for privacy conscious use cases.
Vector databases like Redis to implement semantic search with Retrieval-Augmented Generation (RAG), which gave the LLM a form of memory by allowing it to search a knowledge base for relevant information in response to user queries.
We pulled it all together into a Python web app that would not win any design awards.
RAG really is the technique that makes the whole thing tick. It allows you to provide up to date context not available in the LLM’s training data in a way that doesn’t require fine-tuning. Suddenly, you get an incredibly powerful answer engine that is grounded to your data and is able to quote its sources, much like a peer reviewed paper.
If the idea of having your own local GPT is appealing to you, check out LM Studio.
Building AI Products
AI is such a broad term. Whenever people talk about AI today, they really mean Generative AI, specifically tools like ChatGPT, Claude, MidJourney, DALL-E, etc…
But there is so much more to it, which requires an incredibly diverse set of skills: product management, machine learning, data science, software engineering, platform engineering… as they say, it takes a village. But more important than the technical chops, is the mindset required to solve customer’s pain points.
AI is great but I’ve learned long ago that technology for technology’s sake only takes you so far. We don’t want to be holding a hammer, looking for a nail. We have to start with the problem you are trying to solve and for whom you are solving it. For that, you need people who are curious, eager to experiment, learn, and adapt. This field is evolving so fast that this way of thinking and problem solving is more valuable than relevant expertise trapped in a fixed mindset.
Only then can you recognise opportunities where AI is a key differentiator, able to provide a step change in the business. The Hack[AI]thon demonstrated how powerful this mindset can be — 50 incredible ideas from all across the business. Over 100 passionate individuals obsessed about innovating for our customers.
I’m excited about what lies ahead. Our planning season has kicked off and there is a lot of cool stuff in the cards.
Until next time!