Disclose the size of the hypothesis search on exploratory result displays

Add text that states how many relationships were tested and how narrowly they were preselected. For example, note when one significant hit comes from thousands of screened candidates with only a few expected true relationships instead of presenting the hit as if it came from one planned test.

An exploratory hit looks stronger than it is when readers cannot see how wide the search was.

Mechanism: Showing the size and selectivity of the search makes low pre-study odds visible, which helps readers interpret a significant result as hypothesis-generating rather than automatically credible.

Evidence: The paper states that the greater the number and the lesser the selection of tested relationships, the less likely research findings are to be true, and it gives a 100,000-test example where a significant hit still has very low post-study probability (Ioannidis, 2005).

• User Goal: Judge an exploratory or discovery-oriented claim.
• Task: Interpret a highlighted significant hit from a large search.
• Data: Many tested relationships with only a few expected true ones.
• Chart Setting: A display highlights one or a few significant findings from an exploratory screen.
• Audience: Readers deciding whether the hit is confirmatory or hypothesis-generating.
• Success Criterion: Readers can see the size of the search that produced the hit.

Break it when: The display reports a narrowly preselected confirmatory test rather than a broad exploratory search. Why: The source ties this problem to low pre-study odds created by testing many less-selected hypotheses.

Sacrifice: The headline result looks less definitive. Risk: Readers may treat the result as only exploratory rather than established.

Mistake: Show significant hits from a massive search without reporting how many relationships were tested. Why it fails: It hides the low-prestudy-odds condition that the paper identifies as a major reason significant findings can still be false.

Failure Sign: The display shows a discovery hit but gives no tested-count or expected-true-count context. Quick Check: Ask whether a reader can recover how wide the search was from the display alone. Stronger Test: Add the tested-count and expected-true-count values and inspect whether the implied PPV remains very low.

• Add the total number of tested relationships to the display.
• Add the expected number of true relationships or the assumed R value.
• Mark the result as hypothesis-generating rather than confirmatory.