Artificial Intelligence

The Knowledge Lifecycle of Large Language Models

Authors: Amey Thakur, Sarvesh Talele
Comments: 8 Pages. (Note by viXra Admin: Please submit article written with AI assistance to ai.viXra.org) Code and data: https://github.com/Amey-Thakur/LLM-KNOWLEDGE-LIFECYCLE

A retrieval-augmented system succeeds only when the document it supplies beats what the model already believes. We measure which one wins, at the level of the token the model would actually emit, over 864 forward passes: 6 open models from 124M to 1.5B parameters, 24 counterfactual probes, three phrasings, with and without a contradicting document. Restricting to the 304 cases where the model demonstrably held the fact beforehand, the document is adopted 66.8% of the time and the model keeps its own answer 27.3%. Which model defers is not predicted by size. Holding the fact rises monotonically with parameter count, from 25.0% to 98.6%, but adoption does not: TinyLlama-1.1B holds the fact 97.2% of the time and still adopts the document least of the six, at 38.6%, while Qwen2.5-0.5B, a third its size, adopts 91.8%. Wording moves the result nearly as much as the model does: over all probes the rate spreads by a median of 47.9 percentage points across three paraphrases, and restricted to conflicts, where the spread measures deference rather than recall, it still reaches 57.6. We also show why this has to be read at the rank: the in-context answer carries more than 0.10 of the probability mass in 332 conditions and is still not what the model says in 69 of them, 20.8%. That gap is not hypothetical. It is how an earlier version of this work reached a published conclusion that the measurement did not support, which we correct here. All code, probes and measurements are public.
Category: Artificial Intelligence