In It Was There a Minute Ago, I mentioned that reading research papers had become something of a hobby… It’s borderline masochism. I’ve kept going, working through papers and whitepapers and trying to connect the dots between experiments. What interests me is how a finding in one study changes the way I read another.
Last time, I looked at what a missing answer lets us diagnose. This time, I want to look at the apparent success: the model seems to have learned the material, and we are ready to congratulate ourselves.
A model can become extremely good at predicting the facts in a biography and still struggle when you ask where the person works. Researchers documented that gap. The reading had gone very well. The questions were a bit of a problem.
Think of a pupil who can recite a passage from the textbook, right up to the punctuation. Ask a question that requires taking a fact out of that passage and the performance changes. We would probably want to see the exam results before declaring the revision a success.
That is the distinction I keep pointing to when people talk about getting a brand into model memory. The phrase makes the difficult part sound finished. Once the model has encountered your name during training, it is supposed to know you well enough for the recommendations to follow. Very efficient, if that is all it takes.
There is quite a lot hiding in “knows”.
The lesson came back. The answers didn’t.
Physics of Language Models: Part 3.1 uses fictional biographies to separate memorising text from extracting its facts. The researchers pretrained small models on the biographies, then fine-tuned them on questions about some people and tested questions about the others.
Here, pretraining means learning from the biographies from scratch; fine-tuning is the subsequent training on the question-answer task. Both change the model’s learned parameters, or weights. Information it can use from those weights, without fetching a source, is what we mean here by parametric memory.
The model had seen each test person’s biography, but had never been trained on questions about that person. Failure cannot be waved away as never having received the material.
The pupil comparison helps with the difference between recitation and answering. It tells us nothing about whether a model understands something in the human sense. The question is whether it can use facts learned in one form to answer questions asked in another.
A model that completes your product description, when given the beginning, has passed a particular test. A buyer asking which product suits their needs has set it another. I would be cautious about a memory claim that slides between them without saying so.
The obvious response is more repetition. Give it the lesson again, until it sticks.
In Quantifying Memorization Across Neural Language Models, duplication in the training corpus increases verbatim extraction: supply the beginning of a training passage and the model becomes more likely to reproduce the continuation.
So the comfortable counterclaim, that copies teach a model nothing, is too broad. Repetition can improve a measured result. The question is: which result?
A newer paper, Don’t Forget! Decomposing the Training Dynamics of Memorization in Language Models, asks what happens to a memorised passage as training continues. The researchers used saved versions of Pythia models to simulate training updates, changes to their learned weights.
In those simulations, learning from other data worked more strongly against predicting frequently repeated memorised passages than rare ones, especially in the model with one billion parameters. The researchers suggest that repetition may help keep those passages reproducible despite that interference.
They did not vary repetition and the competing training data separately to prove that explanation. And they tested word-for-word continuation, not answers about the facts.
Coming back to our analogy, the pupil also has the rest of the syllabus to contend with. A mention count cannot tell you how learning from other material affects the result you want.
Being able to finish the passage still leaves open whether the model can use its facts to answer a different kind of question. More successful recitation cannot settle that on its own. Nor is a repeated publication automatically a repeated training exposure. The model provider still has to collect the copy, keep it after filtering and actually use it in training.
I can see why counting mentions appeals. You can count them before you know what they will teach.
The same facts, taught differently
In the biography experiment, the researchers varied the wording, shuffled sentence order and repeated each person’s full name throughout their biography. Accuracy on factual questions about the test group rose from 9.7% to 96.2% in a 124-million-parameter GPT2 model. Those were synthetic facts in a small-model training experiment, not commercial brand recommendations.
The result gives me a better way to think about a brand fact than simply asking how often the sentence appears. Consider an invented payments company. One description explains how its payout feature helps marketplace sellers; another explains how the same feature handles a customer’s payment problem. The descriptions concern the same product, but could help answer different questions.
That gives me a reason to test clear descriptions of a product in different situations. It does not tell me that publishing those descriptions will make a commercial model remember the product better.
The researchers chose their training material and trained the model on it themselves. A publisher can put pages online, but the model provider decides whether those pages enter training, which copies to keep and how to use them.
If you use that result to sell a package of web pages, you are promising an improvement the researchers never tested. Putting the paper in the proposal does not fill that gap.
Better descriptions could still help. I would want evidence that the publishing work improved recall before promising it to a client. And the large gain in this small biography model does not tell us what improvement to expect from a much larger commercial model.
Someone else did the homework
The researchers also kept one group’s biographies unchanged while adding varied biographies for another group. Accuracy on questions about the unchanged group rose from 4.4% to 86.8%.
For the later question-answer training, they used examples about only the added group. The test group’s biographies had not gained extra versions, and its question-answer pairs were never used in training. Training on material about other people had helped the model answer questions about this group. The transfer depended on sufficiently similar material.
Back in the classroom, the pupil gets better at using a lesson because the teaching has improved elsewhere. Their own passage has not acquired another paragraph.
This is the finding that makes a neat brand-content explanation feel incomplete to me. The quality of everything written about a brand may matter. So may the model’s ability to use information of that kind, developed through training on other examples.
If similar transfer matters for commercial facts, your brand could become easier to recall after a model change without anyone having rewritten your brand material. That is a hypothesis about production systems, not a result these researchers measured.
Better answers about your company, given without retrieval, would not necessarily mean your latest content programme taught the model something new about you. The provider may have changed how the model learns to use what it encounters.
I have been tempted to explain better recall through more varied descriptions of each brand. That may help, but it cannot be the only explanation here: the tested biographies did not gain any extra versions. If my explanation needs extra pages about the company, it cannot account for this result.
I like the content-diversity hypothesis. I would rather keep it as a hypothesis than make the paper say it is the whole mechanism.
Better learning still has to meet the question
A separate study, Fine-Tuning or Retrieval?, also finds benefits from paraphrased training material. On its new-fact multiple-choice test, Mistral-7B scored 50.4% after training on the original documents and 58.8% with paraphrased material.
Giving it relevant passages with the question did better: 87.5%. That is retrieval-augmented generation, or RAG, supplying evidence at answer time.
The questions were generated from the same chunks of text that the retrieval system could supply. This favours retrieval: the questions concern facts in the material available to it. The test also provided answer options, so it measured recognition rather than producing an answer without options or recommending a product.
For me, that is another reason to say what the work is supposed to improve. A gain in recognition, a gain in factual answering and a change in product recommendations need different evidence.
The name on a search result can affect the choice too. In Source Preference in the Wild, the researchers kept the titles and snippets fixed, changed the source shown alongside them, and saw the models’ choices change. The models chose from snippets without opening the pages.
The researchers also trained models on pairs of better and worse selections, a form of preference training. They made one source accompany the better item more often during training. Later, the models favoured that source even when the test items met the same measured requirements.
This shows one way training can create a source preference. It does not explain how the search agents acquired their existing preferences. Your product can pass a recall test while a competing result gets picked for a reason the test never examined.
My interest here is commercial too. I work at geoSurge, where model memory is central to the approach. Better recall is an outcome worth pursuing. A publishing recipe that promises it still has to establish the connection between the work performed and the result claimed.
The training experiments give me reasons to take model memory seriously. Changing the material and the training tasks can improve factual answering. That is a useful starting point for further work.
The awkward part is who controls the training. Publishers can make information available and describe it clearly. Providers decide what enters training and how it is used. The biography experiment shows why the material about somebody else can matter too.
If a model starts answering questions about a company better, I would want to understand what changed in the model as well as what changed on the company’s website.
The provider gets to change the curriculum while your content calendar changes the reading material on offer. Confuse those powers and you can keep assigning homework long after the pupil can recite the lesson.