While watching those videos, I had a random thought: instead of building another generic chatbot, what if I try creating a small RAG pipeline that can retrieve insights from actual startup and SaaS marketing transcripts and use that context while generating responses? I wasn’t trying to build some production-ready AI system or compete with existing tools. I simply wanted to understand how embeddings, vector databases, retrieval, and local LLMs actually work together under the hood by building something I personally found interesting. So I decided to experiment with a fully local setup using Node.js, Ollama, Qdrant, Docker, and real YouTube transcript data. This article is basically a documentation of that learning journey — what I tried, what confused me, what unexpectedly worked, what completely broke, and the things I learned while building and experimenting with this project.

Wanting More Than a Generic Chatbot

The main reason I started building this project was because, after using LLMs for a while, I slowly started feeling that most AI answers around startups and SaaS ideas were becoming very repetitive. The responses sounded smart, but they often felt too generalized. Whenever I asked questions related to marketing, positioning, pricing, or validating a product idea, the answers were usually some variation of the same internet advice. It didn’t feel grounded in anything real.

At the same time, I was spending a lot of time watching long-form SaaS and founder-related videos on YouTube. Not short motivational clips, but actual discussions where founders talk honestly about things like customer acquisition, failed experiments, GTM mistakes, distribution problems, pricing struggles, and how they validated their products before building them. Those conversations felt much more valuable because they came from real experiences instead of generic startup advice.

One thing I noticed while watching those videos was that there was a huge amount of useful knowledge hidden inside hours of transcripts, but there was no easy way to interact with it intelligently. Sometimes I would remember hearing a really useful strategy somewhere, but I couldn’t remember which video it came from or the exact context around it.

That’s when I started getting curious about RAG systems.

Initially, my understanding of RAG was honestly very shallow. I thought it was basically just:

But once I started reading more about embeddings, semantic retrieval, and contextual grounding, the whole concept became much more interesting to me. The idea that an LLM could generate completely different answers depending on the retrieved context genuinely fascinated me.

So instead of building another chatbot around PDFs, I decided to experiment with something I personally found interesting — building a small RAG pipeline that could retrieve insights from real SaaS marketing and founder discussions using actual YouTube transcripts. I wasn’t trying to build a production-ready AI startup advisor or some revolutionary system. I simply wanted to understand how these things actually work internally by building one myself and breaking things along the way.

Building the Entire RAG Pipeline Locally

Once I decided that I wanted to experiment with RAG properly, I also knew that I wanted to build everything locally from scratch instead of depending too much on external services or frameworks. A big reason for that is simply because I enjoy building backend systems in Node.js, and whenever I try learning something new, I usually understand it much better when I can see every moving part myself.

So I started keeping the stack very simple and familiar.

For the LLM side, I decided to use Ollama. Honestly, one of the main reasons was how easy its REST API felt to work with. I didn’t want to spend days configuring cloud APIs or dealing with unnecessary setup complexity. I just wanted a local model running on my machine that I could interact with programmatically using Node.js. The moment I saw token streaming working locally through a simple fetch request, it already felt exciting.

For embeddings, I used nomic-embed-text. At that point, I was still trying to fully understand what embeddings actually represented internally. Initially, vectors felt almost abstract to me. Like, I understood the theory that text gets converted into numerical representations, but seeing an actual 768-dimensional vector generated from a piece of text felt strangely fascinating. It was one of those moments where a concept suddenly feels real once you see it working with your own code.

Then came the vector database part.

Instead of going for hosted services or complicated setups, I decided to use Qdrant locally inside Docker. Partly because I genuinely enjoy working with Docker, but also because I wanted the entire system to stay local and lightweight. There was something very satisfying about spinning up a vector database using a single Docker command and then watching embeddings get stored and retrieved from it in real time.

At this point, the overall flow of the system became fairly straightforward. I would first collect YouTube transcripts related to SaaS marketing and founder discussions, clean the transcript data, generate embeddings for the chunks, store those vectors inside Qdrant, and then retrieve the most relevant chunks whenever a user asked a question.

Here’s a simplified architecture diagram of how the entire pipeline worked internally:

But even though the architecture itself sounded simple on paper, the interesting part was actually seeing semantic retrieval work for the first time.

I still remember testing one of the early retrievals and realizing that the system was finding relevant chunks even when the wording of my query was completely different from the original transcript text. That moment genuinely made embeddings “click” for me mentally.

The system wasn’t matching keywords.
It was matching meaning.

And honestly, that was probably the first moment where RAG stopped feeling like another AI buzzword and started feeling like a genuinely powerful idea.

The Chunking Problem and What Completely Changed Retrieval Quality

Initially, I genuinely thought the difficult part of building a RAG system would be the LLM itself. I thought maybe setting up the model, handling inference, or generating embeddings would be the hard part. But surprisingly, the thing that confused me the most was chunking.

And honestly, this was the stage where I slowly started understanding how RAG systems actually work internally.

When I first started experimenting with embeddings, the concept felt very abstract to me. I understood the theoretical definition that embeddings convert text into vectors, but I didn’t really feel what that meant practically. Then after watching the retrievals happen in real time, it slowly started making sense.

The best way I can explain it now is that embeddings are basically numerical representations of meaning.

When we pass text into an embedding model like nomic-embed-text, the model converts that text into a 768-dimensional vector. Obviously, we cannot visualize 768 dimensions, but conceptually similar meanings end up closer together in that vector space.

So , this text :

“How do I validate a SaaS idea?”

and

“How do I know if people actually want my product?”

might use completely different words, but semantically they are talking about almost the same thing. That’s the reason embeddings are so powerful for retrieval systems.

And this is also where vector databases started making sense to me. Traditional databases are extremely good for exact matches and structured queries. But RAG systems are not trying to find exact text matches. They are trying to find semantically similar meaning. That’s why vector databases like Qdrant exist. Instead of searching through raw text directly, they search through vector similarity.

So the overall process becomes something like this :

Once I understood this flow properly, the entire concept of RAG suddenly felt much less “magical” and much more logical.

But then came the real problem.

Initially, I was chunking transcript data very naively. Sometimes fixed-length chunks, sometimes splitting every few paragraphs, sometimes random transcript slicing. Technically, the system was working perfectly. Embeddings were generating correctly, vectors were storing properly, and retrievals were happening successfully.

But the answers still felt weak.

Sometimes the retrieved chunks were only partially related to the question. Sometimes they contained unnecessary filler conversations from the transcript. And surprisingly, there were moments where the base LLM without RAG gave better answers than the RAG-grounded version.

That honestly confused me a lot.

Because in my head, I was assuming:

“More context should automatically improve the answer.”

But slowly I realized something very important:

RAG is not about retrieving more information. It is about retrieving the right information.

A single transcript section may contain storytelling, jokes, filler conversations, random tangents and one valuable insight hidden somewhere inside it. When all of that gets embedded together, the semantic meaning of the chunk becomes noisy.

That’s when I decided to try something different.

Instead of manually creating synthetic question-answer datasets, I literally pasted large sections of YouTube transcripts into DeepSeek and prompted it to generate structured semantic chunks in JSON format.

Something like this:

{
  "chunk_id": 2,
  "videoId": "EsgQ0bQzvZ0",
  "chunk_summary": "The MVP phase focuses on validating the idea before building the actual SaaS product.",
  "content": "Before writing code, founders should first understand the target customer and validate demand through direct conversations and fake sales calls...",
  "questions_on_this_chunk": [
    "How do I validate a SaaS idea before building?",
    "What is fake sales validation?",
    "How do founders validate startup demand?"
  ]
}

And honestly, this changed retrieval quality much more than I expected.

Because now the embeddings were no longer representing random transcript noise. The chunks became semantically focused.

The summaries preserved the core strategic meaning.
The related questions aligned embeddings closer to actual user intent.
And the content itself became much cleaner and context-aware. This was probably one of the biggest learning moments for me during the project. Because this is where I realized that embeddings themselves are not some magical AI feature. The quality of retrieval depends heavily on how intelligently the information is structured before embedding it.

And honestly, that realization also made me understand why production-level RAG systems are much harder than most tutorials make them look.

Things That Surprised me About RAG and LLMs

One of the most interesting things about building this project was how much my understanding of LLMs changed while experimenting with RAG. Before this, I used to think of these models almost like giant systems that somehow “know everything.” But after spending hours testing retrievals, changing prompts, breaking chunking strategies, and comparing outputs, I slowly started realizing that LLMs are actually extremely dependent on the context we provide to them.

I still remember one of the early moments where the retrieval pipeline fetched a highly relevant chunk from a founder discussion related to SaaS pricing. The final response suddenly felt much more practical and grounded compared to the generic answers I was getting before. And honestly, that moment made something click in my head. The model itself was not magically generating business wisdom out of nowhere. It was reasoning using the retrieved context that I had provided to it.

That realization completely changed the way I started thinking about RAG systems. The embeddings were not matching exact words. The vector database was not performing traditional keyword search. The system was trying to find semantic similarity between meanings, retrieve the closest chunks, and then give that information to the LLM as additional context during generation.

But then another thing happened that honestly surprised me even more.

Sometimes the RAG version performed worse than the base model itself. Initially, this made absolutely no sense to me because I kept assuming that more context should automatically produce better answers. But during testing, I noticed that if the retrieved chunks were slightly noisy, partially irrelevant, or overloaded with unnecessary information, the final answer actually became weaker. In some cases, the base model without retrieval gave more direct and better-structured responses, while the RAG version became too constrained by the retrieved context.

That was probably one of the biggest learning moments for me in this entire project.

Most tutorials make RAG look extremely straightforward:

Generate embeddings → retrieve chunks → inject context

But after building it myself, I realized retrieval quality changes everything. A badly structured chunk or slightly irrelevant retrieval does not just reduce answer quality

sometimes it completely changes the direction of the model’s reasoning.

And honestly, this project also made me realize how context-dependent LLMs really are. The models themselves are incredibly powerful, but they are still fundamentally prediction systems. They are very good at reasoning over the information we provide, but the quality of that reasoning heavily depends on the quality of the surrounding context pipeline.

The better the chunking, retrieval, and context formatting became, the better the model started performing. And slowly I started understanding that a huge part of modern AI systems is not just the model itself, but the entire retrieval and context engineering layer around it.

I think that was probably the biggest thing this project taught me. Before building this pipeline, I thought most of the “intelligence” came directly from the LLM. But after experimenting with embeddings, vector retrieval, chunking strategies, and prompt injection, I started realizing that a large part of the intelligence actually comes from how effectively the system retrieves and structures knowledge for the model to reason over.

Final Thoughts

Honestly, this project started mostly from curiosity. I wasn’t trying to build some production-ready AI platform or compete with existing tools. I simply wanted to understand how RAG systems actually work internally instead of only consuming tutorials and theory online. While building this, concepts like embeddings, semantic retrieval, vector databases, and context injection slowly stopped feeling abstract and started making practical sense to me. One thing that surprised me a lot was realizing how much modern AI systems depend not only on the LLM itself, but on the quality of the surrounding retrieval pipeline. The way data is chunked, embedded, retrieved, and injected into the prompt changes the final output significantly. And honestly, building and breaking things myself taught me much more than reading documentation ever could.

This project is still far from perfect, and there’s still a lot I want to improve around retrieval quality, chunking strategies, reranking, and long-term memory systems. But overall, this experiment gave me a much clearer understanding of how real-world RAG systems behave internally and why production-level implementations are much more complex than they initially seem.

If you want to experiment with the project yourself, the entire repository is available on GitHub repo along with the dataset and setup instructions. Everything runs locally using Node.js, Ollama, Docker, and Qdrant, so you can clone the repo and test the full pipeline on your own machine. If you’d like to connect or discuss similar projects, feel free to reach out to me on LinkedIn

Thanks for reading :)