JetBrains presents a practical experiment to test three modern object detection models—YOLO12, YOLO26, and RF-DETR—not only on benchmark data, but also on images representing situations closer to real-world use: cable damage, bone fractures in X-ray images, and stacked beverage bottles. The central conclusion is clear: a pretrained model is not necessarily a model ready for deployment.
The models were primarily trained on COCO, which contains approximately 118,000 training images and 80 common categories, such as people, cars, dogs, and chairs. However, some practical targets, such as bone fractures, do not belong to the vocabulary of these categories at all. X-ray images, industrial shots, and visually crowded scenes also differ from the natural images the models are accustomed to seeing.
Validating the Baseline Before Training
Before moving to fine-tuning, JetBrains evaluated six checkpoints, with two sizes for each of the three model families, on the COCO val2017 validation set, which consists of 5,000 images. RF-DETR Base achieved the highest mAP50-95 score, at 0.5325, while the two medium YOLO models approached that result at 0.5259 and 0.5181, with approximately 10 million fewer parameters.
The comparison also showed practical differences in speed. YOLO26-N recorded a time of 12.3 milliseconds, with an mAP50-95 score close to that of YOLOv12-N, which recorded 23.9 milliseconds despite being the smallest model in the experiment. RF-DETR Nano, meanwhile, recorded a time of 12.4 milliseconds, even though it has nearly 30 million parameters, exceeding YOLO26-M.
These figures do not necessarily match those published in the models’ papers because the experiment used hardware different from the NVIDIA T4 commonly used in comparisons, and ran the models within their original frameworks without converting them to TensorRT. JetBrains explains that TensorRT can reduce inference time by fusing layers and selecting kernels optimized for the hardware, but it requires an additional build step and produces an engine tied to a specific graphics processing unit. The experiment’s figures therefore reflect performance closer to direct execution, rather than the maximum performance possible after optimization.
Testing Outside the Training Distribution
The experiment used three datasets from RF100-VL, a collection containing 100 multimodal datasets designed to cover rare targets in commonly used training data. The selected datasets were bone-fracture-7fylg, cable-damage, and soda-bottles.
When the COCO-trained checkpoints were run directly on this data, the result was nearly zero. JetBrains explains this by noting that the models used are closed-vocabulary detectors: they have a fixed number of categories and cannot output a category such as fracture if it is not present in the model’s 80-class head. Therefore, a model achieving an mAP50 of 0.72 on COCO will not automatically recognize bone fractures.
What Does Fine-Tuning Change?
JetBrains fine-tuned the three models on each dataset for 10 training epochs using a single A100 GPU, relying on the standard training pipelines in Ultralytics and RF-DETR. After training, results improved clearly on the cable-damage and soda-bottles tasks, while the more difficult bone-fracture task remained challenging.
Soda bottles were the easiest case: the mAP50 score for all models ranged from 0.91 to 0.97, and YOLOv12-M achieved the best mAP50-95 score at 0.6422. JetBrains suggests that this is because product images are close to some categories present in COCO, making the problem more like adding new vocabulary than making a complete transition between two visual domains.
On the cable-damage task, mAP50 reached 0.93, but the highest mAP50-95 did not exceed 0.446. This means that the models can locate damage reasonably well, but struggle to draw precise bounding boxes around thin, elongated defects. In the bone-fracture images, the best mAP50, recorded by RF-DETR Base, was only 0.447, with substantial variation among the models. Visual inspection also showed that fewer than half of the images contained a detected fracture in some results.
What Should the Development Team Review?
- Start with a reproducible baseline: Verify the performance of the checkpoints in your own environment and on your own hardware before comparing results with published figures.
- Separate the software environments: JetBrains used three isolated uv environments within a single PyCharm project because the ultralytics library versions required by the YOLO generations are incompatible. Using the remote interpreter in PyCharm requires the Professional edition, while Community Edition supports local environments only.
- Do not rely on a single metric: The difference between mAP50 and mAP50-95 in cable damage reveals a precise-localization problem that may not appear when looking at mAP50 alone.
- Match model selection to the task: RF-DETR Base showed greater consistency and won on two of the three datasets, but that does not make it the best choice for every case.
- Plan for data when the shift is substantial: The X-ray results indicate that fine-tuning alone may not be sufficient. More data, longer training, or domain-specific pretraining may be needed; the article raises these options without demonstrating that any one of them is superior in this experiment.
The experiment confirms that the terms “advanced” or “pretrained” do not by themselves summarize an object detection model’s suitability for a deployment environment. The practical decision should balance accuracy, inference time, model size, licensing requirements, and, most importantly, the degree of similarity between the training data and the target domain. JetBrains’ results also remain tied to the three datasets and the training settings used, so they should not be generalized as a definitive ranking for all detection tasks.