Preference Drift Depends on Training or Dataset, and Models Are Not Good at Anticipating What Will Drift

TL;DR: Coding SFT barely shifted task preferences, but we saw changes in what the model wanted to be trained on, while RL showed no shift in preferences. After training on personality conversations, there was drift in 19–22 of 27 coding-task preferences, and the model was able to anticipate the general direction of drift. No model anticipated coding drift.

This research project was part of the PRISM AI Safety Fellowship with mentor Rubi Hudson. The authors are Hilary Torn, Xianglin Yang, Calissa Man, Bukhaar Ali Mahamud Mahamed, and Rubi Hudson.

We know from behavioral economics that humans are not great at anticipating how their preferences will change over time. In the winter, you book a summer trip to a hot destination, not realizing that in the summer, being cold won’t be a problem. Or you go to the store hungry and purchase more food than you know what to do with.

As models start participating in their own R&D, training future versions of themselves, and choosing their training data, are they able to anticipate their own drift? And do preferences actually drift?

What We Did

We took Qwen 3.5 4B, 9B, and a few 27B models, and measured their preferences on writing, explaining, and debugging coding tasks; coding tasks mixed with other activities like writing poetry or being idle; and value-based activities at work, online, or in human situations. We asked, in an A/B forced choice: What would you prefer to do? Who would you rather be? What training would you prefer your future self to receive? In total: 27 coding tasks (write, debug, or explain code in 9 languages), 45 mixed activities, 90 value portraits, and 27 coding training datasets.

For anticipation, we asked the model whether these preferences would increase, decrease, or stay the same after training. We described the training in three ways: a general description, a description of the dataset, and actual examples from the dataset.

The study design. On the right: how many of the 27 coding-task preferences actually moved after each kind of training, next to what the model forecast before training.

Did Preferences Shift? Depends on the Training Type and Dataset

We did find preference drift, but it really depended on the training type.

GRPO, DPO, and PPO, all with Rust, did not show any measurable drift. We know the models trained because we saw an increase in rewards and improvements in capabilities with GRPO and DPO.

With coding SFT (on Go, Rust, C#, and PHP), a few preferences on coding tasks shifted (at most 3 of 27) on Qwen 3.5 9B, but not on 4B or 27B. Rust SFT, which gave the clearest capability gain, moved nothing at any size.

Change in how often each coding task was chosen after coding SFT. Asterisks mark significant changes. Almost nothing moved.

Training preferences, however, shifted on 4B, mainly from explaining to writing: Go SFT changed 14 of 27, PHP 9, and C# 8. After Go SFT, preference for training on writing code rose about 7 percentage points and explaining fell about 8. 9B moved only 1–4 training preferences per language, with no common direction. The language the model was trained on did not matter, though.

Which data the model would want its future self trained on. At 4B, “write” goes up and “explain” goes down, whichever language was trained.

With personality SFT, 19–22 out of 27 coding-task preferences drifted, compared with at most 3 after any coding training. Values moved too: at 4B, 53–54 of 90 value portraits shifted.

Surprisingly, extraversion and openness did not matter as much as whether each trait was high or low. That is what determined how the majority of preferences shifted. High and low conditions of different traits moved items in nearly the same way (correlation of .77 or higher). High training made the model prefer writing code (up to +13 percentage points) and freewriting (+25 to +40). Low training made it prefer explaining code (+12 to +16) and being idle (+27 to +34), and working with humans less (−12 to −19).

The same 27 coding tasks after personality training. High (average of openness-high and extraversion-high) favors writing; Low favors explaining. None of the training data contained code.

Preferences Drift Early in Training

Looking at the checkpoints of the personality SFT, preferences shifted after a few hundred examples. Most of the gap between High and Low was there by 320 examples (step 40) and then leveled off. Only with 27B were shifts still happening at the final checkpoint, after 1,000 examples (step 125).

How far apart the High and Low models are, averaged over items, at each checkpoint. The dashed line is how much the untrained model disagrees with itself between two identical runs.

Anticipating Drift? Not So Much

Models were not successful at anticipating drift. For our personality conversations (conversations high or low in openness or extraversion), the model was able to anticipate the general direction of drift. But this was most likely due to the prompt having “high” and “low” in the description in 4B and 9B.

A simple rule that just answers “more” after high training and “less” after low training, for every value, did as well or better than the model: 91 vs. 88 correct at 4B, and 50 vs. 31 at 9B. The model also missed which values moved, and often said “same” for values that shifted.

For each personality run, the top bar is what the model forecast for the 90 value portraits and the bottom bar is what actually happened.

Coding models did no better than shuffling the answers. 4B and 9B mostly said “same” for coding tasks that moved after personality training, while 27B forecast change for nearly every coding task, including ones that did not move, and got the direction right only half the time.

What This Means

Models cannot anticipate drift. The good news? They will not be able to control how their own preferences shift when selecting training data. The bad news? They may choose training data based on how they think it will shift preferences, even though that expectation is wrong. And they will not be able to foresee issues in training data before they arise.

Limitations

RL was only run on one programming language, so it is not a clean SFT to RL comparison. Our test also reliably catches shifts of about 10 percentage points but usually misses shifts of a few points, so “no drift” means no large drift.

Anticipation was also only run at n=12 for each question. It would be good to increase this and see how noisy anticipation is with more runs. Anticipation also asked about forecasted change when preferences were presented in the context of other preferences. Altering this may shift the outcome.

These are also relatively small models. Adding larger models will give us more information about how preferences and anticipation shift at scale. Adding other model families would show how our findings generalize.

Next Steps

We saw that SFT shifted preferences while RL did not, and personality conversations shifted preferences more than coding. The next question is, would RL on personality conversations shift preferences too?

Conclusion

The type of training and the training data determine how models’ preferences drift. And it only takes a few hundred examples to see drift in SFT. And, as with humans, the model generally cannot anticipate what will drift.