Recommendations / Research

Recommendations · Researcher

Studying AI at work: methodological pointers for research

Tell a technical promise apart from an observed benefit, and make methods and their limits explicit. Two research settings: AI in professional practice, and the work of the teams who develop and evaluate the models.

By Dr Charles Broutin, occupational physician · Updated on

Read the pathway ↓Open the flowchart →

The essentials

The principle, and what holds in every case.

The steps that depend on no answer.

The principle

“An AI system’s performance is not enough to understand its effects on work and health.”

To do whatever the situation:

  • In all cases. Study the organisation (deadlines, staffing, priorities) rather than employees as a “human factor” to be monitored: it is the organisation that sets the conditions for control.
  • Grade each claim. Established, emerging or unknown. Evidence that is still limited is neither proof of inevitable harm nor proof that there is no risk.

The pathway

Question by question, the recommendation that follows.

The same content as the flowchart, as continuous reading. Each recommendation points to the page of the site it is drawn from.

  1. Start

    Your research is about…

    Branches A and B lead to C, which is common to all studies.

Branch AEvaluating AI in professional practice

  1. A1

    Does your corpus contain health data or data that could identify someone?

    If yes

    De-identify data before anything is sent to a model, and work in an authorised environment (for health data, certified health-data hosting, HDS). To test an online tool, use fictitious texts, like the 540 workplace recommendations in the Préconisations test bench.

    Sources: Publication · AI and medical recommendations · Préconisations tool (in French)

  2. A2

    Are you measuring the model’s performance, or its effects on work?

    A good technical score does not, on its own, predict good workplace use.

    Model performance

    Test on cases representative of the real task: frequent errors and serious errors, stability from one answer to the next, resistance to misleading inputs, cost and turnaround time.

    Effects on work

    Evaluate the work system: time saved including review, final quality, exceptions, responsibilities, workload, autonomy, skills and cooperation, before and after implementation.

    Source: Understanding AI · back to real work

  3. A3

    Are you relying on a published benchmark?

    If yes

    Check that it resembles the target work; describe the instructions, tools, number of attempts and version; specify what the measure captures (accuracy, preference, cost); look for data seen during training and for errors hidden by the average.

    Source: Understanding AI · reading benchmarks

  4. A4

    Is the human reference built independently of the instructions being tested?

    If no

    Build a reference (multidisciplinary consensus, expected rating), tune the instructions on a first sample, then test on a fresh sample. Record the model version, the prompt and the date: each version is a dated object.

    Sources: Publication · AI and medical recommendations · Understanding AI · capabilities and limits

  5. A5

    Are disagreements between the model and the reference re-examined one by one?

    If no

    Classify each discrepancy: justified alert, excessive flagging, missed defect or invention. A disagreement can also reveal a defect the reference had not spotted.

    Source: Publication · AI and medical recommendations

  6. A6

    Do your data on effects rely mainly on self-reports?

    If yes

    Compare them with an observed measure. In a randomised trial by METR, experienced developers believed they were 20% faster with AI; they were measured as 19% slower.

    Source: Understanding AI · perceived and measured gains

  7. A7

    Are the results broken down by task, occupation and population?

    If no

    Break them down: the capability frontier is jagged, and a positive average can hide negative effects. Also look for the organisational changes that explain a favourable or unfavourable effect.

    Sources: Reading · AI and real work · Understanding AI · jagged frontier

Continue to C · concluding and publishing

Branch BStudying those who develop, evaluate or supervise the models

  1. B1

    Does your study setting include control functions?

    Review, pre-release evaluation, red teaming, agent supervision, incident response.

    If yes

    Treat them as jobs in their own right and measure the control workload: volume to review per person, time actually available before a release, staff mobilised during an investigation, on-call duty.

    Source: Reading · Occupational health and AI safety

  2. B2

    Are human approvals used to measure or train the system under study?

    If yes

    Study the state of the supervisors alongside that of the system: a tired supervisor approves faster, and the system can learn to produce what is easily approved. Their health becomes a reliability variable.

    Source: Reading · Human in the loop

  3. B3

    Does your protocol capture the ability to express doubt or to withhold approval?

    If no

    Include it: safety requires that someone can say they are not sure or that things need to wait. An unspoken doubt is a lost safety signal.

    Source: Reading · being able to say no

  4. B4

    Are annotators, evaluators and contractors within your scope?

    If no

    Include them: their judgements are used to train and evaluate the models, but they often fall outside the occupational health monitoring and protections that employees have.

    Source: Reading · Occupational health and AI safety

  5. B5

    Do your data come from the press or from company communications?

    If yes

    Code the testimonies using an explicit grid, such as the six Gollac dimensions in the October 2026 preprint, and draw no causal inference from them. Independent access to deployment data has yet to be obtained.

    Sources: Publication · preprint · Psychosocial risks · state of the evidence

  6. In all cases

    Study the organisation (deadlines, staffing, priorities) rather than employees as a “human factor” to be monitored: it is the organisation that sets the conditions for control.

    Source: Reading · Occupational health and AI safety

Continue to C · concluding and publishing

Branch CFor all studies: concluding and publishing

  1. C1

    Does your conclusion go beyond what your design measures?

    If yes

    Bring it back to what is measured: spotting wording defects does not demonstrate a benefit for health or job retention. State what the study does not measure, and for which model, prompt and corpus it holds.

    Source: Publication · AI and medical recommendations

  2. Grade each claim

    Established, emerging or unknown. Evidence that is still limited is neither proof of inevitable harm nor proof that there is no risk.

    Source: Psychosocial risks · state of the evidence

  3. C2

    Is your question about lasting or causal effects?

    If yes

    Plan longitudinal studies or intervention trials, with independent access to deployment data. Testimonies cannot estimate a prevalence, and neither testimonies nor a cross-sectional study can establish causality.

    Sources: Psychosocial risks · state of the evidence · Reading · AI workers

Publish methods and limits, and repeat the evaluation when the model changes

Taking action

The site’s tools for this audience.

PublicationsA preprint, an INRS study and a narrative review, with methods and limits. Commented readingsArticles, studies and field reports read through psychosocial risk factors. Reading benchmarksTask, protocol, measure, fragility: four questions before concluding.

Key answers

Three frequent questions.

Short answers, drawn from the pathway above.

Is a benchmark enough to assess AI in a professional practice?

No. A good technical score does not, by itself, predict good professional use. Check that the test resembles the intended work, describe the prompt, the tools, the number of attempts and the version, then assess the work system: time saved with review included, final quality, workload, autonomy, skills.

How do you build a human reference to assess a model?

Independently of the prompts tested: a multidisciplinary consensus or an expected rating, prompts adjusted on a first sample, then a test on a fresh sample. The model version, the prompt and the date are recorded: each version is a dated object.

Can the time saving reported by users be trusted?

It has to be set against an observed measurement. In a randomised trial by METR, experienced developers believed they were 20% faster with AI; they were measured 19% slower.

Scope of these pointers

These pointers restate the flowcharts published on this site, which are written for the French legal framework. They are prevention pointers to adapt to each work situation: they replace neither legal advice, nor the opinion of the occupational physician, nor dialogue with employee representatives.

Update log

  1. First continuous-reading version, drawn from the flowchart updated on 13 November 2026.