JetBrains presents a practical experiment testing three modern object detection models—YOLO12, YOLO26, and RF-DETR—not only on benchmark data, but also on images representing situations closer to real-world use: cable damage, bone fractures in X-ray images, and stacked beverage bottles. The main conclusion is clear: a pretrained model is not necessarily a model ready for deployment.
The models were primarily trained on COCO, which contains approximately 118,000 training images and 80 common classes such as people, cars, dogs, and chairs. However, some practical targets, such as bone fractures, do not belong to the vocabulary of these classes at all, while X-ray images, industrial shots, and visually crowded scenes differ from the natural images the models are accustomed to seeing.
Verifying the Baseline Before Training
Before moving to fine-tuning, JetBrains evaluated six checkpoints, two sizes from each of the three model families, on the COCO val2017 validation set, which consists of 5,000 images. RF-DETR Base achieved the highest mAP50-95 score at 0.5325, while the two medium YOLO models came close at 0.5259 and 0.5181, with approximately 10 million fewer parameters.
The comparison also revealed practical differences in speed. YOLO26-N recorded a latency of 12.3 milliseconds, with an mAP50-95 score close to that of YOLOv12-N, which recorded 23.9 milliseconds despite being the smallest model in the experiment. RF-DETR Nano, meanwhile, had a latency of 12.4 milliseconds, even though it has nearly 30 million parameters, exceeding YOLO26-M.
These figures do not necessarily match those published in the model papers because the experiment used hardware different from the NVIDIA T4 commonly used in comparisons, and ran the models within their original frameworks without converting them to TensorRT. JetBrains explains that TensorRT can reduce inference time by fusing layers and selecting kernels optimized for the hardware, but it requires an additional build step and produces an engine tied to a specific graphics processing unit. Therefore, the experiment’s figures reflect performance closer to direct execution, rather than the maximum performance possible after optimization.
Testing Outside the Training Distribution
The experiment used three datasets from RF100-VL, a collection containing 100 multimodal datasets designed to cover rare targets in typical training data. The selected datasets were bone-fracture-7fylg, cable-damage, and soda-bottles.
When the COCO-trained checkpoints were run directly on this data, the result was nearly zero. JetBrains explains this by noting that the models used are closed-vocabulary detectors: they have a fixed number of classes and cannot output a class such as fracture if it is not present in the model head, which consists of 80 classes. Therefore, a model achieving an mAP50 of 0.72 on COCO will not automatically recognize bone fractures.
What Does Fine-Tuning Change?
JetBrains fine-tuned the three models on each dataset for 10 training epochs using a single A100 GPU, relying on the standard training pipelines in Ultralytics and RF-DETR. After training, the results improved clearly on the cable-damage and soda-bottles tasks, while the bone-fracture task remained more difficult.
Soda bottles were the easiest case: the mAP50 score for all models ranged from 0.91 to 0.97, and YOLOv12-M achieved the best mAP50-95 at 0.6422. JetBrains suggests that the reason is the similarity between product images and some classes in COCO, making the problem closer to adding new vocabulary than to a complete transition between two visual domains.
On the cable-damage task, mAP50 reached 0.93, but the highest mAP50-95 did not exceed 0.446. This means that the models can locate damage reasonably well, but struggle to draw precise bounding boxes around thin and elongated defects. In bone-fracture images, the best mAP50, recorded by RF-DETR Base, was only 0.447, with substantial variation between the models. Visual inspection also showed that fewer than half of the images contained a detected fracture in some results.
What Should the Development Team Review?
- Start with a reproducible baseline: Verify the performance of the checkpoints in your own environment and on your own hardware before comparing results with published figures.
- Separate software environments: JetBrains used three isolated uv environments within a single PyCharm project because the ultralytics library versions required by the two YOLO generations are incompatible. Using a remote interpreter in PyCharm requires the Professional edition, while Community Edition supports only local environments.
- Do not rely on a single metric: The gap between mAP50 and mAP50-95 in cable damage reveals a precise-localization problem that may not appear when looking only at mAP50.
- Match the model choice to the task: RF-DETR Base showed greater consistency and won on two of the three datasets, but that does not make it the best choice for every case.
- Plan for data when the domain shift is large: The X-ray results indicate that fine-tuning alone may not be sufficient. More data, longer training, or domain-specific pretraining may be needed; the article presents these as options without demonstrating the superiority of any one of them in this experiment.
The experiment confirms that the labels “advanced” or “pretrained” do not summarize an object detection model’s suitability for a deployment environment. The practical decision should balance accuracy, inference time, model size, licensing requirements, and, most importantly, the similarity between the training data and the target domain. JetBrains’ results also remain tied to the three datasets and the training settings used, so they should not be generalized as a definitive ranking for all detection tasks.