Fine-tune a model on 'A is B' and it stops answering 'B is A', even when the answer is in the prompt.
- role
- Sole author
- status
- Research
- when
- 2026
- ownership
- Solo
Who did what. Independent research. Paper, code and the SynthRel-500 dataset are mine.
Overview
Forward-Only Fine-Tuning Induces a Reversal Asymmetry in GPT-2. I fine-tuned GPT-2 Small, Medium and Large on forward-only questions over a synthetic dataset and tested them on held-out facts whose statement is always in the prompt. Every model answers forward questions perfectly but fails many reverse ones, and every wrong reverse answer is either empty or the forward answer: the model has learned a directional answer rule, not lost the association.
Why it exists
The Reversal Curse is usually studied for facts stored in the weights. I wanted to know what fine-tuning in one direction does when the fact is right there in context, why a few demonstrations repair it, why a single demonstration can make things worse, and which parts of the network change.
My part
- Built SynthRel-500: 500 relational facts across five relations over procedurally generated names, split 70/15/15 so test facts never appear in fine-tuning.
- Ran twelve fine-tuning runs (three scales, four seeds) with exact McNemar tests and bootstrap confidence intervals, and classified every wrong answer.
- Ran activation patching, direct logit attribution and head and MLP ablations with TransformerLens, mostly on GPT-2 Medium, then compared across scales.
- Audited the work against the released code during revision and reported the implementation issues I found, including a label-masking error that left the first answer token unsupervised, with what each one affects.
How it works
- a
Dataset
SynthRel-500: invented entities, five relations, fixed splits.
- b
Fine-tune
Forward-only questions; the fact is always in the prompt.
- c
Measure
Forward vs reverse, zero-shot and with k = 1, 2, 4, 8 demonstrations.
- d
Inspect
Patching and direct logit attribution locate where demonstrations act.
- e
Ablate
Remove top heads and MLP blocks to test what retrieval needs.
Results
| Model | SFT forward | SFT reverse | Reverse, 4 seeds | k = 1 | k = 4 |
|---|---|---|---|---|---|
| GPT-2 Small 117M | 100% | 9.33% | 4.00 to 12.00% | 0.89% | 95.56% |
| GPT-2 Medium 345M | 100% | 64.00% | 41.33 to 64.00% | 30.22% | 98.67% |
| GPT-2 Large 774M | 100% | 40.00% | 2.67 to 40.00% | 75.11% | 100.00% |
A rule, not a gap
Of 658 wrong reverse answers across twelve runs, 423 were the forward answer and 235 were empty. Not one named any other entity.
One example hurts
A single demonstration drops Small and Medium below zero-shot; the errors mostly copy an entity from the demonstration.
Where it enters
Patching shows demonstration information reaching the final position between layers 11 and 16 in Medium, plateauing at +3.97 logits.
Heads and MLPs
Ablating the top ten heads costs 10 to 20 points in four Medium runs; ablating MLP layers 6 to 18 removes demonstration-based reversal in Medium and Large.
More forward training data (35 to 350 examples) does not remove the asymmetry. During revision I checked every analysis against the released code and report the issues I found, the most important being a label-masking error that left the first answer token unsupervised, together with which conclusions each one affects.
Built with
- Python
- PyTorch
- TransformerLens
- GPT-2
- Pandas
- LaTeX
- Reverse accuracy after SFT (S / M / L)
- 9.33% / 64.00% / 40.00%
- Forward accuracy after SFT
- 100% at every scale
- With 4 demonstrations
- 95.56% to 100%
- Wrong reverse answers (12 runs)
- 658, all empty or the forward answer
Status & next
A single demonstration drops reverse accuracy below zero-shot in Small (9.33% to 0.89%) and Medium (64.00% to 30.22%), mostly by copying an entity from the demonstration. Demonstration information reaches the final position around layers 11 to 16 in Medium, and ablating the ten highest-attribution heads lowers demonstration-based reverse accuracy by 10 to 20 points across four Medium runs.
Source: the paper (Khanna, Chitkara University) and github.com/AA-KH/Reversal-Curse-Mechanisms