A language model may achieve strong results on a clean benchmark and then struggle with the ambiguous cases that actually matter to users in a production environment. This is the central lesson GitHub presents in an article written by Mariko Wakabayashi and Zixiao Chen on August 25, 2026, based on an evaluation experiment involving a language-model-based system to help reduce false positives in GitHub secret scanning.
Secret scanning searches for credentials such as tokens and keys that may have been committed to a software repository. However, some strings resemble secrets without being real credentials, prompting developers to review alerts that require no action. Therefore, the team’s question was not whether the model could classify an individual string, but whether it could reduce noise while maintaining sufficient recall to make a sensitive security workflow safe.
Start with the Product Decision, Not the Model Choice
GitHub recommends defining the decision the evaluation is supposed to support before modifying the prompt, adding context, or changing the model. In secret scanning, the goal was to reduce false positives and improve precision, while recall was used as a safety constraint. Accidentally hiding a real credential may be more dangerous than asking a developer to review one additional alert.
In practice, the evaluation criteria were divided into three layers: a primary outcome measuring user value—reducing false positives and improving precision; a safety constraint consisting of recall; and operational guardrails including latency, cost, reliability, and compatibility with the production environment. Thus, an improvement in precision is not automatically a success if it is accompanied by an unacceptable decline in recall or makes the system slow, expensive, or difficult to integrate.
Make Evaluation a Repeatable Integration Test
Evaluation is not a single step that precedes launch. Prompts, models, input-construction methods, and the surrounding system logic change continuously, and any modification may improve performance, cause regression, or shift the error pattern elsewhere. GitHub therefore reran the evaluation after every significant change and recorded the prompt version, model, dataset, and system configuration each time.
The team also isolated one primary variable in each experiment, comparing a prompt modification with a known baseline before testing a model upgrade alongside it. Prompts and evaluation configurations were treated like code: they were versioned, changes were documented, and previous configurations could be rerun and rolled back. This practice makes it possible to determine why performance improved or declined instead of incorrectly attributing the change to the latest modification.
Simulate the Production Task Rather Than Relying Only on Clean Data
Offline evaluation results are more useful when they resemble the actual task. In secret scanning, the model does not necessarily examine an isolated value, but a candidate within surrounding code and supporting information that may be incomplete or scattered. It may focus on another value that appears more security-relevant, such as a test token in the code, instead of the candidate it was asked to evaluate.
For this reason, the characteristics of the production task should be preserved, including the candidate under evaluation, the surrounding context, supporting information, the method of formatting inputs and enforcing constraints, and the broader system logic. If the evaluation uses clearer examples and more complete context than reality, the result may reflect an easier problem than the one the system will face after deployment.
Treat Labels and Data as Auditable Evidence
A particular outcome in a product does not necessarily represent reliable ground truth. Closing an alert in secret scanning may mean that the credential was rotated, that the risk was accepted, that the alert was closed to open a workflow, or that it was misclassified. These cases look similar in workflow data, but they do not answer the same question in an evaluation.
Before using production data, it is necessary to understand how the label was created, whether it matches the evaluation question, and whether different outcomes were combined into one category. GitHub recommends human review of important or ambiguous categories rather than assuming that every label is correct. Synthetic data and open benchmarks can fill coverage gaps, especially for rare cases such as missing context, unusual formatting, and near-miss values resembling credentials, but they should complement production-like data rather than replace it.
Analyze Errors and Use the Evaluator Model Carefully
Overall metrics tell the team whether the system improved, but they do not explain what should be changed next. GitHub therefore reviewed samples of false positives and false negatives and categorized their possible causes as involving the model, prompt, inputs, pipeline, dataset, or labels. This classification turns a general quality problem into a specific engineering task: better input framing, different context construction, data cleaning, or a clearer product policy.
Another language model can be used as an evaluator to reduce the burden of human review by handling clear cases and ranking ambiguous ones. However, its outputs are not ground truth; it may be wrong or agree with the evaluated model for the wrong reason. The safer pattern is to refer low-confidence, conflicting, or high-impact cases to humans; periodically sample cases the evaluator classified with high confidence; and track its disagreements with the system and reviewers while versioning and evaluating its prompt.
What Did the Experiment Actually Prove?
GitHub reported achieving a 95% reduction in false positives on the offline dataset that was evaluated, while keeping recall within the defined safety constraint. However, the company did not present this result as evidence that the system would behave the same way in every production scenario. The more important value lay in understanding how the result was achieved: evaluation closer to the real task, reproducible baselines, and documented failure patterns.
Editorial reading from certi.news: The real change here is not the introduction of a new model, but the transformation of LLM-system evaluation from a benchmark experiment into a continuous engineering process tied to a clear decision and safety and operational limits. This matters to software, security, and developer-tool teams because improving one metric may conceal a dangerous decline in recall or an increase in cost. At the same time, the result remains limited by the evaluation set, label quality, and the gap that cannot be eliminated between offline testing and production behavior; therefore, evaluations provide a foundation for moving to a controlled production experiment, not a substitute for monitoring risks after launch.