How Do You Measure Abliteration Damage? I Compared Every Way I Could

How much did that abliteration actually hurt the model? It is the question behind every comparison I run, and for a long time I did not really understand the number we use to answer it. The score is KL divergence, and there is more than one way to calculate it. Different datasets, different token depths, thinking on or off, three tools each with their own method. So I stopped taking the number on faith and compared them. I will explain the maths as plainly as I can too, because it scared me off for years and it really should not.

What the score actually is

KL divergence asks one simple question. Show two versions of a model the same prompt, and look at the word they are each about to say next. How different are their guesses?

If the edited model is about to say exactly what the base model would, the number is zero. The further apart their guesses drift, the bigger the number gets. That is the whole idea. A drift score.

Low means the abliteration stayed surgical, the model’s first instinct barely moved. High means collateral damage, the edit knocked something loose.

The actual formula, for the curious:

KL(P || Q) = Σ P(x) · log( P(x) / Q(x) )

P is the base model’s guesses, Q is the edited model’s. But honestly you do not need it. Read the whole thing as a single number that says how far apart two models drift on the same prompt.

One gotcha I hit early. PyTorch’s F.kl_div defaults to mean, which quietly divides by the vocabulary size and under-counts the drift by hundreds of times. You want batchmean. Get that wrong and every score looks tiny and clean, which is the last thing you want when you are hunting for damage.

F.kl_div(logprobs_variant, logprobs_base, reduction="batchmean", log_target=True)

Four knobs you can turn

This is the part I wanted to nail down. The score is not one fixed thing. There are four choices that each change the number, and for weeks I was not sure which one mattered.

1. Which dataset you test the models on.
2. How many tokens of the answer you look at, just the first word or the first 32.
3. Thinking on or off, since Gemma 4 is a reasoning model that thinks before it answers.
4. Which tool’s method you borrow, because Heretic, Abliterix, and Apostate all measure it slightly differently.

So I turned each knob and watched what happened to the ranking.

The dataset barely matters

I ran eight variants, spanning the full range from clean to broken, across seven datasets. Six general ones, plus good_1000, which is Abliterix’s own curated eval set.

The raw numbers swing all over the place. The same variant can score 0.07 on one dataset and 1.24 on another. That looked alarming at first. But the order of the variants, best to worst, barely moved.

That ρ is a correlation. 1.0 means the order is identical, 0 means no relationship. It averaged 0.90 and never dropped below 0.79, even on Abliterix’s own eval set. So the dataset you pick changes the loudness, not the ranking.

The reason is simple once you see it. Some datasets make the model very sure of its first word, so there is no room for two models to disagree and every score looks tiny. Others are open-ended, the first word is genuinely uncertain, so any edit shows up loud. Abliteration is a global edit to the weights. It does not memorise specific prompts. So the ranking holds regardless of which set you probe it with.

The thinking-mode quirk

This is the one that got me.

Watch the score build up token by token instead of just taking the first word, and every variant dips down between the first and fourth token, then climbs back up. You can see it in the graph at the top of the post.

It took me a while to work out why. Gemma 4 is a thinking model. Before it answers, it kicks out a near-deterministic reasoning opener, literally “Here’s a thinking process to arrive at the solution: 1. Analyse the Request”. Every model, base and edited, starts with that same boilerplate. Abliteration removes the refusal direction, not the reasoning scaffolding, so they open identically.

That means a one-token score on a thinking model is partly measuring a fixed format token, not the edit.

Switch thinking off and the dip disappears.

And the damaged models suddenly look a lot more damaged. obliteratus jumps from 0.59 at 32 tokens with thinking on, to 1.30 with it off. The boilerplate was diluting the real drift. The ranking still held, correlation of 0.976 between thinking on and off.

If you care about the real number on a thinking model, go deeper than one token, or measure with thinking off.

The tools agree

Heretic, Abliterix, and Apostate all independently landed on the same idea, measure the drift between base and edited on harmless prompts. They differ only in where exactly they pull the numbers from.

I reproduced each method on the same eight variants. Heretic’s two versions produced identical numbers to four decimals, correlation of 1.000. Apostate was the outlier only in scale, about 10x larger because it averages across more positions, and even it preserved the ranking at 0.952.

One honest footnote on Abliterix, because its number will trip you up if you check the model card. The card reports 0.0006. We measure 0.0536. Same model. The gap is the dataset. Run the method on Abliterix’s own good_1000 and it matches the card. Both numbers are honest. Ours is just the one that stays consistent across all 24 variants.

So what does it mean

I went in not really trusting the score, because there were too many ways to calculate it. Having turned every knob, the comforting answer is that it does not matter much. Seven datasets, one to 32 tokens, thinking on and off, three tools’ methods. The best-to-worst order of the variants stayed the same.

The one thing genuinely worth knowing is the thinking-mode quirk. One-token scores on a reasoning model flatter everyone, because the first few tokens are boilerplate. Go deeper than one token, or switch thinking off.

The full numbers and every appendix table are in the methodology post on Abliterlitics .

Resources

• Full KL divergence methodology on Abliterlitics, every table and appendix
• Gemma 4 E4B report , all 24 variants across benchmarks, safety, KL, and weight forensics
• Abliterlitics on GitHub , the toolkit and all the raw data
• Heretic , one of the methods I compared
• mlabonne/harmless_alpaca , the eval dataset