How to Avoid the Biggest Bias in Your Own Research: Confirmation Bias

How to Avoid the Biggest Bias in Your Own Research

Confirmation bias is the OSINT practitioner's most expensive failure. It is the tendency to seek, weigh, and remember evidence that supports your current best guess, while under-weighing or forgetting evidence that contradicts it. It is the default state of a person who has spent two weeks on a project. The five moves below are the ones that catch it before it costs you a brief.

The bias is not a moral failure

Confirmation bias is not a moral failing. It is the cognitive default for anyone who has been working on a single project for more than a few days. The way to handle it is to design the project so the bias is visible on the page, not to pretend you have eliminated it.

Tip

Move 1: Pre-register a falsification

Before you start a project, write down the specific kind of evidence that, if found, would change your current best guess. Two reasons this works:

• It forces you to imagine a world in which your conclusion is wrong. That is the most important check on overconfidence.
• It gives you a stopping rule. When the falsification is met, you stop and re-plan. When it is not met, you can publish with more confidence.

A pre-registration that says "any primary source from the company confirming the launch was on track for Q2 would change my conclusion" is much more useful than a pre-registration that says "I might be wrong about this." The specific version forces you to commit, on the page, to a world in which your conclusion is wrong.

The falsification belongs in the brief, not in your private notes

A falsification that lives in your private notes is a falsification that does no work. Put it in the brief. A reader who disagrees with your conclusion can point to the falsification and say "this is the part you are missing." A reader who agrees can check that the falsification has not been met.

Warning

Move 2: Show the negative evidence

A brief that lists only the records that supported the conclusion is a press release. A brief that lists the records that did not support the conclusion is an audit trail. The negative evidence is the part the reader is going to want to see, and the part you are most likely to leave out.

A useful habit: in the write stage, before publishing, build a list of the records you found that did not fit the conclusion. For each, one line: what the record said, why it did not fit, and what you did with it (excluded, weighted down, surfaced in the contradicts list).

Three patterns to watch for:

• The silent exclusion. A record that did not fit the conclusion and was quietly left out of the brief. The silent exclusion is the most expensive form of confirmation bias.
• The wrong weight. A record that did not fit the conclusion and was given a low weight in the brief, when it deserved a high one. The wrong weight is harder to spot than the silent exclusion but just as expensive.
• The "but" sentence. A sentence in the brief that says "while source A is consistent with the conclusion, source B suggests otherwise." The "but" sentence is doing the work of the negative evidence; make sure it is doing the work explicitly, not in passing.

Move 3: Have a hostile reader

Before publishing, give the brief to someone whose job is to find the holes. The first round of "but what about…" is the most useful feedback you will get on a research project.

A few rules for the hostile reader:

• They do not have to be an expert. The point of the hostile reader is to surface the gaps a non-expert would notice. A domain expert will see the technical gaps; a non-expert will see the structural ones.
• They should read the brief, not the records. The brief is the artifact under review. The records are the source. If the brief does not stand on its own, the records will not save it.
• They should not be you. Self-criticism is real, but it is not a substitute for a second reader. A second reader does not share your investment in the conclusion.
• Their feedback should be on the page. A brief that responds to the hostile reader's questions is more useful than a brief that ignores them.

A useful practice: before you send the brief, write down the three questions you would ask if you were the hostile reader. If the brief does not answer them, the brief is not done.

Move 4: Time-box the project

A project that has been running for a year is rarely the place where the next clean insight lives. The sunk-cost fallacy — the tendency to keep going on a project because you have already invested time in it — is the close cousin of confirmation bias. A short, scoped project is more often a publishable unit than a long one.

A few rules for time-boxing:

• Set a deadline before you start. A project with no deadline is a project that runs forever.
• Treat the deadline as a hard line. A project that needs another week probably needed a better plan, not more time.
• Publish at the deadline, even if the brief is not perfect. A short, honest brief is more useful than a long, perfect one that never ships. The next project will catch what this one missed.

The exception is a project where the cost of being wrong is high. A brief on a financial decision, a legal claim, or a public-interest finding should be allowed more time than a brief on a personal curiosity. The time-box scales with the stakes.

Move 5: Name the bias on the page

A brief that says "I might be wrong about this" is more useful than one that does not. The act of naming the bias is the act of acknowledging that you, the researcher, are a source of error in the project. The acknowledgment is what makes the rest of the brief auditable.

A few patterns:

• The corpus biases paragraph. A short paragraph in the methods section naming the three biggest biases of the corpus and what you did about them. Standard in any serious brief.
• The reader biases paragraph. A short paragraph naming the biases the researcher is most likely to bring to the project. "I have been writing about this company for three years, which likely makes me over-weight the company's positive announcements and under-weight the regulator's enforcement notices."
• The "what would change my mind" paragraph. A short paragraph at the end of the brief naming the specific kind of evidence that, if found, would change the conclusion. The act of writing it forces you to imagine a world in which your conclusion is wrong.

The biases do not go away when you name them. But a brief that names them is a brief that lets the reader correct for them.

The move that catches the most misses

A single move is good. All five together is the practice. The most expensive research mistakes happen when the project runs long, the writer is the only reader, and the brief lists only the evidence that supported the conclusion. The five moves above are the discipline that prevents that.

The discipline is not a one-time fix. It is a practice. Every project you run on this rhythm is a project where confirmation bias has fewer places to hide.