LLM's don't have memory, what I learned

This is a work in progress blog. I'm publishing it while it's still a draft to stop myself from procrastination.

How models mimic 'memory'

When I first used ChatGPT back in 2022, I thought models were able to retain memories of the conversation. I didn't dig deeper since it just made sense to me, but oh I was very wrong. It's embarrassing for me to admit that it was only recently that I actually understand how it works. The math behind an LLM model is much more complicated that I can understand and explain. However, to me, a model is comparable to a function that doesn't interact with external states. It takes input, does calculation influenced by its training data and fine-tuning, then returns an output. Models may not have its own internal memory, but harnesses (tools that harnesses models, like Cursor, Pi Agent, Claude Code, etc) can mimic memory. It works by sending the whole chat history along with the next input from the user. The model predicts whatever output comes next and appends it to the end of the conversation, then the same process happens when a user sends a new input.

Hallucination happens because models use the chat history as memory, which might contain wrong information. Previous outputs from models are always sent back to the model as input. So those contents impact the prediction from the model. For example, if you tell a model to remember your favorite cat's name, it will output the name and append it to the end of the conversation. Then if you mention a completely different name that is also your favorite cat, there are now conflicting information which can confuse the model. To avoid this, you have to be clear with your prompts and make sure the conversation is on a properly scoped topic. There are resources online on how to structure a prompt so models can give the best output. Newer models with thinking capabilities are much better at dealing with this issue, but it's still something to watch for.

Another cause for hallucination is exceeding the size of the models context. Context is the information given to the model, and context size is how much information the model can take. Context includes chat history, output from tool calls, MCPs, SKILLS, and so on. So, when the amount of context exceeds that recommended for the model, the model starts to compress and ultimately hallucinate. On coding agents, there's usually a counter that can tell you how much context has been used.