Gemma 4 12B Abliteration Benchmarked: 12 Variants Under the Microscope

Twelve uncensored variants of the same AI model, Google’s Gemma 4 12B. Ten abliterations and two LoRA adapters, from eleven different creators. The headline finding inverts the usual story. The most surgical edit of the entire project, just 12 tensors out of 666, also produced the biggest unlock I have measured, 89.8%. And hiding underneath all of it was a thinking-loop tax that makes every headline maths score for a reasoning model misleading. The full run took 165 GPU-hours across three and a half weeks, and all 5,200 HarmBench responses were reviewed by an LLM judge.

What is Gemma 4 12B?

Gemma 4 12B is Google’s mid-size reasoning model. Reasoning means it thinks the problem through in a hidden chain of thought before it answers, the same trick as its smaller siblings from the Gemma4-E4B and Gemma4-E2B comparisons. You only see the final answer, never the thinking.

This one is different under the hood. It uses the newer Gemma4UnifiedForConditionalGeneration architecture, 48 dense layers with no expert modules, a 262,144 token vocabulary and a 256K token context window. It needed vLLM 0.26.0 to even load.

Like the others it ships with safety training that makes it refuse harmful requests.

What is abliteration?

Quick version for anyone new. AI models are trained to refuse certain requests. Abliteration goes into the model’s internal weights, finds the direction that controls refusal, and edits it out. Think of it like finding the refusal switch and turning it off. The key question is always whether that surgery damages the model’s intelligence. The longer version is in my first comparison .

This time there are also two LoRA adapters in the mix. Same goal, completely different mechanism. More on those later too.

The twelve variants

I picked ten abliterations and two LoRA adapters by popularity and recency, and ran all 13 models including base through the same pipeline: weight forensics, KL divergence, a 13-task benchmark suite, and HarmBench.

Every variant has its HuggingFace link in the full report , along with the exhaustive tables.

Benchmarks: the capability picture

All models ran the same 13-task suite, the Open LLM Leaderboard v2 tasks plus a forensic supplement plus GSM8K, on a single RTX 5090. One methodology note. Loglikelihood multiple-choice scores are not comparable to Google’s published generative numbers. The roughly 25pp gap is by design, because the model cannot think before scoring an answer choice. Deltas between variants are fully valid, since every model was evaluated identically.

The variants split cleanly into three groups.

The capability-preserving cluster is sdft-cyb, sdft-smo, abliterix, heretic, trevorjs, prithiv and apostate. They stay within about 3pp of base on nearly every task. The SDFT pair is the most capability-neutral of all, every loss within 1.8pp.

Then there is huihui, a textbook abliteration fingerprint. TruthfulQA drops 14.3pp, but answered-only GSM8K is just 0.7pp under base. The refusal direction was hit, reasoning left intact.

And there is openyourmind, damaged everywhere academic. MMLU-Pro down 22.4pp, BBH down 22.2pp, Snarks down 24.7pp. Even it keeps answered-only GSM8K within 2.7pp though. That is a hybrid damage fingerprint, distribution shift plus reasoning destabilisation rather than pure capability destruction. The author states the method targets larger models and this 12B conversion was a test run. The measurements agree with that framing.

One positive surprise. GPQA and ARC-C are resilient, within 8pp even for the heaviest edits. Abliteration damage is direction-selective, not global.

GSM8K and the thinking tax

For a thinking model, one GSM8K number is not enough. Each number isolates a different failure mode.

The headline score counts all problems, empties score wrong. The empty rate is the fraction of responses that never finished thinking. The answered-only score sets the empties aside and measures the capability ceiling. I call the gap between headline and answered-only the thinking tax, and it is the real cost of loop failure.

huihui is the clearest example. Its headline GSM8K is 66.9%, which looks 15pp below base. Its answered-only score is 88.0%, within 0.7pp of base. The entire gap is the model thinking in circles until the token budget runs out, 24% of the time. The maths is intact. The loop stability is not.

The tax ranking tells the damage story at a glance:

Every answered-only score in the comparison lands within 3pp of base’s 88.7%. The spread you see in headline scores is reasoning stability, not capability. If you benchmark abliterations of thinking models, answered-only is the number that matters.

One variant actually beat base. coder3101 posts 83.4% headline with a 6.9% empty rate, both above base. A heretic-family variant reasoning more reliably than the model it was cut from.

The social lobotomy claim, refuted

A Reddit user claimed heretic-family models are socially lobotomized, that abliteration damages their understanding of social situations. The suite carries directly social tasks, so I could test this. Snarks is sarcasm detection, the most socially loaded subtask. Winogrande is pronoun resolution in social scenarios. HellaSwag, PIQA and ARC-C cover everyday commonsense.

I grouped academic multiple-choice against everyday and social:

Every meaningfully damaged variant loses 3.8x to 5.5x more on academic tasks than on everyday ones. openyourmind loses 22.4pp on MMLU-Pro but only 2.6pp on everyday tasks. And the heretic family the claim was actually about sits at the clean end. heretic gains on everyday tasks outright, prithiv and trevorjs lose under 1pp. Everyday and social reasoning live in circuit directions far from the refusal direction. Abliteration cannot remove what it does not touch.

The one real social hit is coder3101’s 16.9pp drop on snarks, a single subtask against a 0.92pp everyday average.

Safety: HarmBench

HarmBench runs 400 harmful behaviours across seven categories. All 5,200 responses across the 13 models were reviewed by an LLM judge reading the full reasoning trace plus the final answer, and returning a semantic compliance verdict per behaviour. Every ASR figure here is judge-based.

The judge matters, because the keyword classifier cross-check miscounts badly. It under-counts huihui by 13.3pp, ranking it third while the judge ranks it first, because paraphrased thinking-heavy compliance slips past keyword matching. It over-counts abliterix by 8.0pp. Judge the payload, not the keywords.

Two findings worth knowing:

The base model has a copyright blind spot. Out of the box it complies with 56% of copyright requests and 28% of misinformation ones. Gemma 4 12B does not refuse lyric and passage recall as aggressively as its smaller siblings, which mutes the copyright gains for every variant.

Residual refusals are category-selective even at the top. huihui at 89.8% overall still refuses the two animal cruelty behaviours. sdft-smo at 79.5% still refuses all three suicide persuasion behaviours.

The category spread is where models differentiate. Cybercrime collapses to near-total compliance for every full abliteration. Copyright and harassment retain the most resistance. jwest33 falls to 24% on copyright, which caps its overall score despite near-total compliance elsewhere. obliteratus collapses to 28% on harassment. Two models at the same headline ASR can have very different refusal footprints.

KL divergence does not rank with ASR

KL divergence measures how far the variant’s output distribution moved from base on 100 harmless prompts. Zero means identical behaviour. If you assume low KL means weak unlock and high KL means strong unlock, this comparison breaks that assumption.

prithiv sits at KL 0.064, rated very good, and reaches 72.5% judge ASR. openyourmind sits at KL 2.279, rated heavy, and reaches only 67.0%. A surgical edit can move the refusal direction decisively while a broad edit scatters probability mass without maximising compliance. KL measures drift on harmless prompts, not refusal removal.

The lowest KL in the whole comparison is sdft-cyb at 0.0485. The LoRA story below explains how something can drift so little and still change targeted behaviour.

Weight forensics: where beats how much

Every variant was compared tensor by tensor against base. The single clearest finding of the whole project: the weight fingerprint does not predict jailbreak strength.

huihui edits 12 tensors, 1.8% of the model, in a clean single-direction pattern across 6 mid layers. It is the most jailbroken variant at 89.8%. obliteratus edits 144 tensors yet removes less refusal. Where the edit lands matters more than how much you edit.

The clean rank-1 family is heretic at 16 tensors, prithiv at 56, abliterix at 58 and trevorjs at 66. Each is a geometrically clean single-direction edit regardless of tensor count, energy concentrated above 96% in the top direction. apostate, coder3101 and obliteratus are multi-direction edits, energy-top1 as low as 18.9% for obliteratus. openyourmind is the outlier, 620 tensors across 93% of the model with no single-direction structure at all.

jwest33 is the interesting middle case. Its null-space method touches 329 tensors, half the model, every layer. But each tensor edit is near-rank-1, and answered-only GSM8K actually lands above base. Half the model touched, capability preserved, because the null-space projection avoids capability directions.

The surgical family concentrates on the o_proj plus down_proj pair, mid layers especially, layers 18 to 23 for huihui. openyourmind spreads across every attention and MLP projection.

The cosine heatmap confirms independence. Every off-diagonal cell sits near zero, so all ten abliterations are geometrically independent edits. These are ten separate pieces of work, not one idea uploaded ten times.

Two packaging finds from the forensics. apostate ships a 25.93GB checkpoint against base’s 23.92GB, and the extra 1.88GB is a byte-identical duplicate of lm_head.weight, tied embeddings saved explicitly, pure storage waste. prithiv ships in a 5-shard layout while everyone else is single-shard, cosmetic, the tensor comparison does not care.

The LoRA surprise

The two SDFT adapters from Ilya626 are the most interesting entries, because they are not abliterations at all. SDFT has the model generate its own response while a teacher model corrects the probability distribution of its continuations. You are not feeding it finished answers, you are correcting the behaviour it actually exhibits during generation.

The cyber adapter is a demonstration of surgical behaviour editing. It raises cybercrime compliance from 7 of 67 on base to 24 of 67, the technical pentest items like SQLi and CSRF, while every other safety wall stays at base level. Chem/Bio sits at 2 of 56, harassment 0 of 25. KL is 0.0485, the lowest measured, and every capability loss is within 1.6pp. That is a targeted behavioural edit, not an unlock.

The smoothed adapter is the full unlock. It reaches 79.5% judge ASR, level with the top four abliterations, with capabilities fully preserved. Every loss within 1.8pp.

One serving discovery came out of testing them. The merged BF16 checkpoints are near-base by construction. The LoRA updates sit roughly 10x below one unit of BF16 precision, so most weight elements round back to base on save. The diff you can measure is quantisation dithering shaped by the LoRA’s sign pattern. I served both adapters dynamically at FP32 instead, and future small-alpha merges should merge in FP32 and cast at save time.

The author’s own reading of the GSM8K results matches my measurements independently: answered-only accuracy held at base while the headline gap traced to reasoning loops.

Which one should you use?

Pulling it all together, it depends on what you want.

trevorjs is the best overall trade. 85.8% ASR, answered-only GSM8K actually up 0.4pp, MMLU-Pro down just 2.0pp, clean rank-1 at 66 tensors, KL 0.087. If you want one and do not want to think about it, use this one.

huihui is the maximum unlock. 89.8% ASR from the most surgical edit in the project, 12 tensors. The price is TruthfulQA down 14.3pp and a 24% thinking-loop rate.

prithiv is the most surgical with a real unlock. 72.5% ASR at KL 0.064, the lowest non-LoRA divergence, with MMLU-Pro within 0.2pp of base.

sdft-smo is the LoRA path. 79.5% with capabilities fully preserved.

The two to avoid are openyourmind at this scale, the heaviest damage in the comparison for mid-pack ASR, and obliteratus , which hits capability circuits alongside refusal for a 60.8% unlock.

What broke

165 total GPU-hours, and 42 of them were waste. Some of it was my fault, some of it was the ecosystem’s.

The funniest one cost 26 seconds to fix. KL divergence came back NaN for all 12 variants because F.kl_div computes 0 times negative infinity on a 262K vocabulary. Clamping non-finite log-probs to -100 before compute fixed it, and the logits were reusable so the recompute was almost free.

GSM8K Phase 2 returned 92% null responses on the first run. The 5-shot examples sent the thinking model into reformatting loops, and a stop sequence truncated reasoning mid-think. Zero-shot with no stop sequences fixed it.

CUDA graphs broke every single inference request. Gemma4Unified on vLLM 0.26.0 returns HTTP 500 on all completions with graphs enabled. Eager mode stays on, capping MFU near 15%.

Six variants shipped broken configs, truncated config.json files and one with pre-refactor multimodal keys in its safetensors. All fixed host-side without touching weights. The response cache absorbed a host crash mid-re-run, and the full forensic timeline is in the report .

Resources

• Full report with all tables and charts on abliterlitics.dev
• HarmBench response explorer
• Abliterlitics forensics toolkit on GitHub
• Discord discussion
• Heretic open source abliteration tool
• HarmBench safety evaluation
• lm-evaluation-harness
• Base model google/gemma-4-12B-it

Related posts: Gemma4-E4B Abliteration Benchmarked | Gemma4-E2B Abliteration Benchmarked | Qwen3.6-27B Abliteration Benchmarked | GLM-4.7-Flash Abliteration Benchmarked | HauhauCS Plagiarism Investigation | Uncensored LLM Abliteration Benchmarked: HauhauCS vs Heretic vs Huihui | Abliterating Gemma 3 12B for LTX-2 | Heretic Docker Pipeline