Domain validity
Questions must sample the actual knowledge and reasoning required in the field rather than merely reuse generic medical examinations.
Thomas Cole, The Consummation of Empire, 1836
III. EvaluateEvaluate promises before believing them.
Evaluate promises before believing them.
A spectacular demo is not evidence.
AI systems, work organisation and occupational health.
A generally capable model may still fail on a specialised task. Evaluation must be contextualised, reproducible and kept separate from marketing claims.
In 30 seconds
Essential points
- General medical or multilingual benchmarks do not test the specific legal and institutional knowledge of French occupational health.
- A reproducible benchmark reports repeated runs, exact prompts, model versions, missing answers, costs and uncertainty.
- Detailed results from the specialised benchmark are intentionally withheld pending publication of the scientific article.
EstablishedEmergingAnalysis
On this page
This page summarises the method. The French edition contains the fuller protocol discussion.
Rationale
Why French occupational health needs a dedicated benchmark
Occupational health in France combines medicine, prevention, employment law, social-security tables, institutional roles and highly specific professional practices. A model can perform well on general clinical questions while confusing the employer’s duties, the occupational physician’s role or the legal conditions of fitness decisions.
French context
Language is not the only issue. Regulations, institutions, terminology and professional responsibilities are jurisdiction-specific.
Reasoning tasks
Clinical vignettes and workplace scenarios are needed to test whether the model can apply rules to context, not only recall isolated facts.
Transparency
Publication status
The benchmark article is awaiting publication. Until the manuscript and methods are publicly accessible, this site does not publish a model ranking or headline results. This avoids presenting unreviewed numbers as established findings and reduces the risk of a score being reused without its methodological limits.
What is available
The rationale, evaluation principles and reasons for using a specialised corpus.
What remains withheld
Model-level results, rankings and inferential comparisons until the article is public.
What will matter later
The exact model versions, evaluation dates and reproducible protocol because the model landscape changes quickly.
Protocol
What a useful benchmark should report
Evaluation quality depends on details that are often omitted from marketing claims. The corpus should be documented, answer keys reviewed, prompts frozen, repetitions justified and errors inspected by category.
Corpus
Number of items, domains covered, source dates, reasoning versus recall, single versus multiple answers and validation procedure.
Execution
Provider, model identifier, date, temperature, token limits, tools, retries, missing answers and total API calls.
Statistics
Confidence intervals, paired comparisons when appropriate, effect sizes, multiple-testing correction and sensitivity analyses.
Operational measures
Latency, price, refusal rate, answer stability and the consequences of different error types.
Decision-making
How to interpret benchmark results
The best average score is not automatically the best deployment choice. A model may be marginally more accurate but far more expensive, unstable or difficult to host securely. Error severity also matters: a rare but consequential legal or medical error can outweigh many minor wording improvements.
Avoid rank worship
Small score differences may fall within uncertainty or depend on a narrow subset of questions.
Inspect errors
Look for systematic confusion about legal roles, contraindications, prevention hierarchy or data confidentiality.
Re-test locally
A public benchmark is a screening tool. The intended workflow still requires local validation and post-deployment monitoring.
Key references
- NIST, AI Risk Management Framework.
- The complete methodological discussion and future publication details are maintained in the French reference edition.