Pair uncertainty numbers with study-validity notes

Add brief validity notes next to the estimate instead of showing the number alone. For example, pair a risk or effect value with study size, duration, inclusion criteria, and a compact note or profile covering major threats to internal validity, external validity, or bias.

A number alone does not tell readers whether uncertainty comes from normal variability, weak study design, limited relevance to their situation, or a weak scientific foundation. Visible validity notes reduce the need for readers to guess about those hidden threats.

Mechanism: Showing where uncertainty comes from helps readers calibrate trust in the estimate instead of overtrusting a precise-looking number or inventing their own explanation for its weakness.

Evidence: The paper argues that observers otherwise must infer uncertainty for themselves, shows bias-audit summaries as a useful profile of methodological flaws, and notes that readers can extract needed information from formats that combine numerical summaries with narrative summaries of threats to validity (Fischhoff & Davis, 2014).

Notes: The paper’s protocol organizes these notes around variability, internal validity, external validity, strength of science, and a resulting credible interval.

• User Goal: Decide whether to trust an estimate enough to compare options or act on it.
• Data: The estimate comes from studies or models with known limits in sampling, measurement, bias, or transfer to other settings.
• Chart Setting: The display already shows a summarized number, interval, or recommendation and has room for a caption, side note, or boxed summary.
• Audience: Readers are not expected to infer study flaws on their own.
• Success Criterion: Readers can state both the estimate and at least one important limitation affecting it.

Break it when: Observed variability is the only meaningful source of uncertainty and threats to internal validity, external validity, and scientific pedigree are negligible. Why: The paper treats those added threats as the reason simple confidence-style summaries are not enough.

Sacrifice: You use more space and attention than a number-only display. Risk: Too many caveats can overwhelm readers. Mitigation: Use a standard compact set of fields rather than ad hoc prose.

Mistake: Showing only the estimate or interval with no note on sample, duration, eligibility, bias, or relevance. Why it fails: Readers either overtrust the number or have to guess about important hidden threats.

Failure Sign: There is no place on the display that names why the estimate might be less certain than it looks. Quick Check: Can a reviewer locate notes on study size, duration, inclusion criteria, bias risk, or relevance to the target setting? Stronger Test: Ask intended readers what could make the estimate less trustworthy; if they cannot answer from the display, the validity notes are missing or too vague.

• Add a compact note listing study size, duration, and inclusion criteria next to the estimate.
• Add a brief bias profile that marks major sources as high, low, or unknown risk.
• Add a note on how far the study setting or population matches the reader’s target setting.
• Expand a simple confidence-style summary into a credible-interval summary when these added threats materially affect the result.