bartowski published a new experiment on the calibration data used to calculate imatrix for GGUF models, explaining that the new dataset is not a “magic solution” for improving performance. However, it produced limited yet meaningful gains in some cases and provided a deeper understanding of how the type of text used in quantization affects the final model. The experiment’s author says this is not the final result and that he expects to continue updating and testing the data.
The experiments were conducted in collaboration with Fable and using GPU resources provided by LTT Labs. bartowski publicly released the files used, including two separate files for prose and conversations, along with the texts after they had been passed through the model’s chat template. He also provided a script explaining how to generate the formatted version and noted that the Qwen files are available with deliberate additions of <|endoftext|> strings at the end so the model does not ignore the final part of the data.
What function does imatrix serve in the quantization process?
bartowski explained that the quantization algorithm in llama.cpp attempts to minimize the error resulting from converting groups of weights into lower-precision representations. To estimate the relative importance of these weights, large quantities of text are passed through the model, after which measurements of input-channel activations are collected—more precisely, the sum of the squares of the activations reaching each tensor.
A channel that activates strongly and consistently across many texts is treated as indicating that the weights associated with it are more important to the final result. A channel that rarely activates may appear less important, while there remains a possibility that the data did not include the right type of text to trigger it. Therefore, the result depends not only on the size of the corpus, but also on its diversity and ability to activate the model’s different components.
The tests covered dense and mixture-of-experts models
The team tested seven models: gemma-4-E2B-it, Qwen3.5-4B, Qwen3.6-27B, Qwen3.6-35B-A3B, Mistral-Small-4-119B, Qwen3-Next-80B-A3B, and Qwen3.5-397B-A17B. The evaluation focused on comparing KLD with bf16, testing BFCL, running a set of short custom tests, and checking expert coverage in MoE models. Samples from MMLU-Pro and GSM8K were also used for deeper verification because of the cost of running the complete tests.
The general result was that the data did not produce a clearly expected difference at rates above approximately 4 bits per weight. The early tests included completely random data to rule out certain hypotheses, but the differences in Q4_0, IQ4_XS, and higher levels remained limited in most cases because quantization errors at these levels are relatively small.
The largest difference appeared at lower levels, particularly Q2_K in MoE models. When Qwen3.6-35B was quantized without imatrix, BFCL performance fell by about 28 points, from approximately 82% to 54%. The difference between the best and worst corpus also reached about 10% in this test. By contrast, most of the clean data used with IQ2_M was similar, with differences generally not exceeding a few percentage points.
Why do chat formatting and tool use matter?
The experiment indicates that the challenge in MoE models is related not only to language diversity or text quality, but also to the data’s ability to activate different experts. In the Qwen3-Next-80B-A3B model, an English corpus dedicated to tool use left 18 out of 512 experts without any activation, even after 3126 segments had been processed—nearly four times the length of calibration_datav5.txt.
Combining calibration_datav5.txt with tool-calling data achieved complete expert coverage. The experiments found that multilingual and programming content were among the main factors needed to activate some of these experts. Including conversations formatted according to a chat template also appeared to be particularly important in MoE models, because some experts may activate only when they encounter chat tokens and structure.
In practice, this means that preparing an imatrix for a model that supports chat or tool calling should not be limited to random prose. The absence of a chat template or tool examples may cause some experts to be underrepresented during calibration, which may later appear in performance at low quantization rates.
How was the new version built?
bartowski used segment-level analysis to determine which parts of the data activated experts most effectively. The results showed that some experts appeared only with specific types of content, such as code, French text, fictional literature, or scientific material. Diverse prose was therefore collected with the aim of maximizing expert coverage across the MoE models used in the test.
Tool-calling conversations from interstellarninja/hermes_reasoning_tool_use were added to this, and a 2:3 ratio between prose and conversations was considered the best combination in the tests. After testing versions ranging in size from 400 to 800 segments, the final version performed better than much larger versions, even when those versions were built on the same quality. The author offers an initial explanation that the most common content in a huge corpus may overwhelm important but less frequent signals.
He also said that the new data did not reduce multilingual coverage; rather, the share of text written in the Latin alphabet decreased in the prose portion, meaning that non-Latin languages represented a larger share of the corpus compared with version v5, while emphasizing that additional tests are still required.
The comparison between v5 and v6 is not an absolute victory
bartowski compared versions v5 and v6 when building Qwen3.8-27B, using perplexity and KLD on wikitext, HuggingFaceH4/no_robots, and Salesforce/xlam-function-calling-60k. He describes these figures as approximate indicators rather than conclusive evidence, particularly because the bf16 reference perplexity value on no_robots was 19.05. On wikitext, average KLD improved with v6 at some levels, such as Q2_K_L by 2.6% and IQ2_M by 1.8%, but declined at others, such as Q4_K_S by 3.6% and IQ2_S by 3.1%.
This comparison shows that the gains from the new dataset are selective rather than a general upgrade for every quantization format or every test set. This supports the author’s conclusion that the effect of imatrix is more apparent at low bit rates and in models that rely on distributing work among a large number of experts.
The author also tested using a context length of 2048 instead of 512 when calculating imatrix, but did not observe improved results; expert coverage actually declined in many cases. A cosine-similarity comparison between imatrix results calculated from bf16 and Q8_0 also showed similarity exceeding 99.99%, with some outlier values.
The practical conclusion is not that a single dataset will always guarantee better performance, but that corpus selection has become a factor worth tuning, especially when quantizing MoE models to low levels. The new version is available with its files and generation method, while bartowski continues testing it on other models, including Mistral and Qwen3-Next-80B-A3B.