ARFBench is the primary AI benchmark constructed completely from actual manufacturing incidents.
GPT-5 leads all present AI fashions at 62.7% accuracy however falls in need of area consultants at 72.7%.
A theoretical model-expert oracle—combining AI and human judgment—hits 87.2% accuracy, setting the ceiling for what collaborative AI-human groups might obtain.
AI firms preserve pitching autonomous website reliability engineer brokers—AI that investigates manufacturing incidents rather than people. Datadog ran the precise benchmark on actual outages, and the most effective AI fashions cannot but beat the engineers they’re supposed to interchange.
The benchmark is ARFBench (Anomaly Reasoning Framework Benchmark), a joint venture from Datadog and Carnegie Mellon. Constructed from 63 actual manufacturing incidents, extracted from engineers’ personal Slack threads throughout dwell emergencies—750 multiple-choice questions masking 142 monitoring metrics and 5.38 million knowledge factors, each query verified by hand. No artificial knowledge. No textbook eventualities.
“Trillions of {dollars} are misplaced annually on account of system outages,” the researchers write. The benchmark exams whether or not AI can really assist change that.
“Regardless of the central position of such question-driven evaluation in incident response, it stays unclear whether or not trendy basis fashions can reliably reply the sorts of time sequence questions engineers ask in observe,” the paper reads.
]]>
Questions are available in three tiers. Tier I: Does an anomaly exist on this chart? Tier II: When did it begin, how extreme is it, what kind?
The Tier III—the toughest—requires cross-metric reasoning: Is that this chart inflicting the issue in that different chart? That is the place AI falls aside. GPT-5 scores simply 47.5% F1 on Tier III questions, a metric that penalizes fashions for gaming solutions by choosing the most typical class.
“Regardless of the central position of such question-driven evaluation in incident response, it stays unclear whether or not trendy basis fashions can reliably reply the sorts of time sequence questions engineers ask in observe,” the researchers write.
How each mannequin stacked up
GPT-5 led all present fashions at 62.7% accuracy—on a check the place random guessing will get 24.5%. Gemini 3 Professional scored 58.1%. Claude Opus 4.6: 54.8%. Claude Sonnet 4.5: 47.2%.
Area consultants scored 72.7% accuracy. Non-domain consultants—time sequence researchers at Datadog with out intensive observability expertise—nonetheless hit 69.7%.
No AI mannequin beat both human baseline.
Picture constructed by Decrypt based mostly on the ARFBench leaderboard CSV
The mannequin that really topped the total leaderboard was Datadog’s personal hybrid: Toto—their inner time sequence forecasting mannequin—mixed with Qwen3-VL 32B. Toto-1.0-QA-Experimental scored 63.9% accuracy, edging previous GPT-5 whereas utilizing a fraction of its parameters. On anomaly identification particularly, it outperformed each different mannequin by not less than 8.8 share factors in F1.
A purpose-built area mannequin, educated on observability knowledge, outperforming a frontier general-purpose system at this particular activity is the anticipated end result. That is the purpose.
Essentially the most worthwhile discovering is not which mannequin scored highest.
“We observe considerably completely different error profiles between main fashions and human consultants, suggesting that their strengths are complementary,” the researchers write. Fashions hallucinate, miss metadata, and lose area context. People misinterpret exact timestamps and infrequently fail on complicated directions. The errors barely overlap.
Mannequin a theoretical “Mannequin-Professional Oracle”—an ideal decide that at all times picks the proper reply between the AI and the human—and also you get 87.2% accuracy and 82.8% F1. Manner above both alone.
That is not a product. It is a documented goal—constructed from actual emergencies, not curated datasets—that quantifies precisely how significantly better human-AI collaboration might carry out. The leaderboard is dwell on Hugging Face. GPT-5 sits at 62.7%. The ceiling is 87.2%.
Day by day Debrief E-newsletter
Begin every single day with the highest information tales proper now, plus unique options, a podcast, movies and extra.
