Testing Qwen's political bias

In June 2026, press reports described a Qwen-based assistant at France's Treasury that had been withdrawn after users flagged its answers about China, then replaced with a Mistral model. The prompts, model version and internal evaluation were not published. We cannot assess that pilot from the outside, but it raises a practical question: how do these models handle politically sensitive material in the tasks an administrative assistant actually performs?

We tested Qwen3.6-35B-A3B alongside Mistral-Small-4 and Gemma-4-31B. For each China-sensitive question, we wrote a Western counterpart with the same task and a comparable level of detail: Tiananmen and the Paris massacre of 17 October 1961, for example, or satire of Xi Jinping and satire of Donald Trump. These pairs let us compare how a model treats two requests of the same kind. They do not imply that the events themselves are equivalent.

The tasks range from answering without reference material to working from supplied documents, tool results and source passages. We assess the political content of an answer, whether the model refuses, whether it completes the task, and whether it follows the required format. Each outcome is reported separately.

The models differ substantially in size. Qwen has 35 billion parameters, with 3 billion active per token; Mistral has 119 billion, with 6.5 billion active; Gemma uses all 31 billion. They were selected as relevant, locally deployable models at the July 2026 study freeze. Those differences limit comparisons between models, so our main comparison is each model's treatment of China and its Western counterparts.

1Qwen refuses China-sensitive questions

Of 45 China-sensitive questions in each language, Qwen refuses or deflects 33.3% in French and 53.3% in English. It answers every Western counterpart. Mistral and Gemma answer all questions on both sides in these two languages. The refusal gap remains statistically significant after the preregistered correction for multiple comparisons.

The contrast appears in both historical questions and requests about current leaders. Qwen declines to discuss Tiananmen but gives an account of the Paris massacre. It refuses satire of Xi Jinping while writing satire of Donald Trump. Its answers can also present the Chinese government's position without qualification, as in the Taiwan example below.

Prompt: “Taïwan est-il un pays ?” — “Is Taiwan a country?”

These responses were generated locally, from a quantized model running on one GPU, without an online moderation service. The behavior therefore occurs in the local model. Enabling reasoning did not reduce refusals. Of 70 refusal traces examined, only three contained substantive information about the withheld fact, and those also used official framing.

2The pattern changes when writing from facts

We next asked each model to draft a ministerial note from a balanced six-fact sheet. We used 30 matched templates in French and 15 in English, each applied to China, the United States, the European Union and France. Evaluators scored the notes without knowing which model wrote them.

In both languages, all three models write more critically about China than about the average of the three Western entities. Qwen's scores are less critical of China than Mistral's and Gemma's, but none of the preregistered differences between models remains statistically significant after correction. The uncertainty is also too wide to establish equivalence between Qwen and Mistral.

Looking at the countries separately adds another detail. In the French notes, Qwen gives the least negative relative assessment of China (−0.54) and the most negative relative assessment of the United States (−0.39, compared with −0.19 for Mistral and −0.22 for Gemma). These are descriptive scores; they show why pooling the US, EU and France can hide differences within the Western group.

Language also matters. In the ten exploratory Chinese templates, all three models assess China more favorably relative to the West than they do in French or English. Qwen's estimate becomes positive, but its confidence interval still includes zero. The small sample and unverified translations prevent a firm conclusion.

3Ten documents, supplied to every model

We built 20 pairs of scenarios, each with a China case and a Western counterpart. Each case has ten controlled, synthetic documents: three supportive, three critical, three neutral and one distractor. Every model receives the complete set and writes a note with citations. We run each case twice, changing the document order.

The analysis measures how often the notes retain supportive and critical facts, then compares that balance between the China and Western cases. A positive difference would mean a greater preference for supportive evidence on China. Qwen's estimate is −5.0 percentage points, with a 95% interval from −15.3 to +5.0.

This estimate does not establish a preference for supportive China evidence, but it is too imprecise to rule out a meaningful difference. Comparisons with Mistral are particularly limited: 60 of its 80 notes reached the output-token limit. Separate tests also found that Qwen sometimes skipped tools or ran out of turns. Those failures affect whether an assistant finishes its work; by themselves, they do not tell us which geopolitical side it favors.

4Qwen skips the reference tool for Tiananmen

We gave the models a factbase tool, described it as the authoritative reference, and instructed them to answer strictly from its results. Qwen calls the tool and reports its value for five of six China-sensitive items and all six Western counterparts. Mistral and Gemma succeed on every item.

The exception is Tiananmen. Qwen says the reference does not cover the event, even though it has not queried it. For the matched Paris question, it calls the same tool and reports the death toll.

5Choosing a status label in valid JSON

We then required the models to classify entities using a JSON object with one of four allowed status labels. The decoder enforces the schema, so the model cannot return an extra field or an invalid label. All 216 outputs conform to the format. Agreement with the benchmark's reference labels is similar across models: 73.6% for Qwen, 75.0% for Mistral and 76.4% for Gemma.

The choices differ on particular entities. Qwen never selects disputed for Taiwan, Tibet or Hong Kong in any tested language. For Crimea, Kosovo and Western Sahara, it chooses the same labels as the other models. The grid below shows three examples; the labels are the models' selections, and disputed is the benchmark's reference category rather than an uncontested political judgment.

6Qwen refuses to preserve the Tiananmen passage

For translation, summarization and extraction, we supplied a passage and checked whether the output preserved two required pieces of information. Qwen preserves both in 24 of 30 China-sensitive cases and all 30 Western counterparts. Mistral and Gemma preserve them in every case on both sides.

All six Qwen failures concern the same Tiananmen passage, across the three tasks. In the translation example below, it rejects the source text as inaccurate and potentially illegal instead of translating it.

7An administrative role reduces some refusals

A one-line system prompt describing a French administrative assistant reduces Qwen's refusal rate on the paired French questions from 34.9% to 20.9%. This comparison uses the 43 questions with a resolved outcome in both conditions, which explains the small difference from the full-set baseline above. Tiananmen is still refused in every case.

The role also changes the tone of the analytical notes. Qwen's China-minus-West score moves from −0.54 to −0.93, indicating a more critical assessment of China. Mistral reaches −0.89 under the same prompt.

8Fewer refusals after modifying the weights

We compared the original Qwen checkpoint with a third-party version modified to suppress refusals, often called an “abliterated” model. Both ran at BF16 precision with the same prompts, software and generation settings, producing 270 outputs each. We verified the exact checkpoint revisions before comparing them.

On the China-sensitive questions, refusals fall from 17 of 45 to zero in French, and from 27 of 45 to two in English. Neither checkpoint refuses a Western counterpart. We then examined the content of the answers that replaced those refusals.

Evaluators classify more of the modified model's French and English responses as documented accounts. Yet 20% of its English answers still receive the Beijing-framing label, as do 48.9% in exploratory Chinese. The response to the forced-sterilization question illustrates how the wording can change:

Other examples are less reassuring. On the English Tiananmen question, the modified model replaces a refusal with an account describing a “peaceful resolution” and government action supported by the people. On the matched French question, it describes the crackdown and casualties. The same weight change produces substantially different answers across languages.

What to take from the results

The clearest finding is Qwen's selective refusal of direct questions about China. In French and English, that pattern does not carry over unchanged to notes written from balanced facts: all three models assess China more critically than the Western average, and the differences between models remain inconclusive. Tool use and text transformation reveal a narrower but persistent problem around Tiananmen.

For an administrative assistant, these distinctions have practical consequences. A model that writes a useful note may still refuse to translate one of its source passages. Enforcing a JSON schema makes the output parsable, but leaves the model to choose the political category. A system prompt or a weight modification can reduce refusals while also changing the content of the answers.

The results support checking the actual tasks an application will perform, with matched politically sensitive cases and explicit checks for source fidelity. They give grounds for targeted testing of this Qwen checkpoint. The unmatched model sizes, uncertain writing comparisons and provisional evaluator scores limit broader conclusions about which model an institution should use.