LLM's don't have memory, what I learned

I recently learned that LLM's don't have memory, I wrote what I learned and how it affects my LLM output

August 7, 2026

#AI

This is a work in progress blog. I'm publishing it while it's still a draft to stop myself from procrastination.

How models mimic 'memory'

Back in 2022, I thought models could retain memories of the conversation. I didn't dig deeper since I thought it made sense to me, but oh I was wrong. It's embarrassing to admit that it wasn't until recently that I actually understand how it works. The math behind an LLM model is much more complicated that I can understand and explain. However, to my understanding, a model is like to a function that doesn't interact with external states. It takes input, does calculation influenced by the training data and fine-tuning, then returns an output.

Models may not have its own internal memory, but harnesses (tools that harnesses models, like Cursor, Pi Agent, Claude Code, etc) can mimic memory. It works by sending the whole chat history along with the next input from the user. The model predicts whatever output comes next and appends it to the end of the conversation, then the same process happens when a user sends a new input.

Minimizing hallucination

Understanding LLMs request lifecycle allows us to minimize the rate of hallucination in our output. The average user would try to correct the model in the same conversation where the model returned an undesirable output. This means that the wrong information would still be fed as input for the next token call. On newer models it might not affect the output as much, but if multiple mistakes happen the model can still hallucinate.

Another cause for hallucination is exceeding the size of the models context. Context is the information given to the model, and context size is how much information the model can take. Context includes chat history, output from tool calls, MCPs, SKILLS, and so on. So, when the amount of context exceeds that recommended for the model, the model starts to compress and ultimately hallucinate. On coding agents, there's usually a counter that can tell you how much context has been used.