Open Source Intelligence

Open source intelligence is the practice of producing honest, sourced insight from information that is already public. The phrase was first used inside the U.S. intelligence community in the late 1980s, but the discipline has long since escaped that origin. Today, OSINT is the working method behind investigative journalism, fact-checking, due-diligence reports, competitive research, academic literature reviews, and a surprising amount of academic study by curious teenagers.

It helps to fix a few terms before we go further.

What counts as an open source

An open source is any information that is available to the general public without breaking the law. The medium does not matter. It includes:

• Web pages, blog posts, and forum threads
• Public social media accounts
• Government filings, court records, and corporate registries
• Academic papers and conference proceedings
• Books, magazines, and broadcast archives
• Public datasets, satellite imagery, and open sensor feeds
• Leaked documents that are already in wide circulation

It does not include private messages, paywalled databases you do not have a subscription to, hacked material, or anything that requires deception to obtain. The openness is what makes the resulting insight defensible. A claim that rests on a stolen inbox looks exciting on a blog and crumbles the first time a serious reader asks to see the chain of custody.

The simplest test

If you can show a stranger the exact URL, document, or dataset your claim comes from — and they could read it themselves tomorrow — you are doing OSINT. If you cannot, you are doing something else.

Tip

What it is not

OSINT is often confused with a few adjacent practices. They overlap, but they are not the same thing.

Most real projects sit at a seam between two or three of these. A reporter writing about shell companies will use OSINT to map ownership, data journalism to crunch filings, and traditional reporting to talk to people. The OSINT is the part that anyone could, in principle, replicate from open material.

The shape of an OSINT question

Almost every OSINT project starts with a question that is more specific than it looks. "What is X doing?" is not a question. "What shipments has X made to ports in country Y in the last 18 months, and how does that compare to public statements by X about its supply chain?" is a question — it implies a corpus, a method, and a falsifiable answer.

The chapters in this guide treat that gap as the main thing to fix. The technology — search engines, scripts, language models — is downstream of the question. The question is the lever that pulls the rest into place.

OSINT is a verb, not a noun

The point is not the collection. The point is the brief you can hand to someone else that says, with sources, what you actually found.

Info

The intelligence cycle is the oldest diagram in the field and still the most useful. It is a loop, not a list, because every step changes what the next step is for. The version below is a practical one, written for a person running a one-person project on a laptop.

The six steps

1. Plan and frame

Write the question down in one sentence. Then write down:

• What would change my mind. A specific kind of evidence that, if found, would falsify your current best guess.
• What is in scope. The geographies, time window, and source families you will work with.
• What is out of scope. Topics you will not chase, even if you trip over them.

If you cannot fill those three blanks, you are not yet doing OSINT — you are still browsing.

2. Collect with intent

Pick surfaces before you pick queries. A surface is a single source family: a site, a database, a search engine, a forum. For each surface, write the query you will run, and the date you ran it. Pages change. Without a timestamp, your future self cannot reproduce the work.

Save the query, not just the result

The URL is half the record. The query, the date, and the source you ran it against are the other half. Storing only the answer is the most common beginner mistake.

Tip

3. Process into records

Raw collection is a mess: PDFs, screenshots, half-loaded pages, social posts that may be deleted by morning. Turn each piece into a record with the same shape: a title, a source URL, an archive URL if you have one, the date you captured it, and a one-line description in your own words.

The point is not bureaucracy. The point is that a record you cannot find next month is the same as no record at all.

4. Analyze for patterns and tensions

Now read. Look for repetition across independent sources, and look for disagreements that may be doing real work — the gap between a company's press release and a regulator's filing is a finding, not a contradiction. Resist the urge to declare a winner too early. Most open-source stories are the story of someone who did.

5. Write the brief

A one-page brief, in plain language, with inline links to your records. A reader who has never heard of your topic should be able to follow the chain and disagree with your conclusion on the basis of the same evidence you saw.

6. Re-plan

The brief is also where the next loop starts. Every reader question, every gap, every "but what about…" is a new entry in the planning step. The cycle is short — serious projects loop through it a dozen times in a week.

Where the loop tends to break

The two failure modes are almost always at the same seams:

• Skipping planning. You start collecting, the collection feels productive, and the brief is forced into whatever shape the collection happens to support. This is how you end up with a 4,000-word post that does not answer the question you actually had.
• Skipping the write step until the end. The write step is not a report. It is the first place the analysis has to commit. If you cannot write a sentence about what a source means, you have not yet finished the analyze step.

The write step is the analysis

A common claim is that the analysis happens earlier. In practice, the writing is where most of the real thinking gets done. If you are avoiding it, the project is avoiding you.

Warning

Most OSINT failures are not collection failures. They are corpus failures. The researcher downloaded 4,000 pages, then wrote a paragraph that only used 12 of them — usually the 12 that came up first on Google. The other 3,988 were decoration.

The fix is to treat the material you gather as data, not as reading. That means giving it a schema, sampling it honestly, and being honest about the gaps.

Pick a format before you collect

For a typical project you will end up mixing three formats. The right ratio depends on the question.

A useful default: a single spreadsheet of records, a notes document that you write into as you go, and a folder of raw captures linked from the records. Resist the urge to live in any one of these.

The schema is the argument

The columns you choose for your records encode what you think matters. A spreadsheet of "URL, date, notes" treats every source as interchangeable. A spreadsheet of "URL, date, claim, speaker, source type, confidence" forces you to commit to what each source is doing in your argument. That is the point.

For a project about a company's public statements, the schema might be:

• id — a stable identifier
• date_published — when the source first said it
• speaker — who is on the record
• claim — the actual sentence you are tracking
• source_type — filing, press release, interview, social post
• archive_url — a Wayback Machine snapshot when you have one
• confidence — your own rating, kept private

You will throw the schema away and build a new one halfway through. That is fine. The first schema is the move that makes the second one possible.

Sample honestly

If you cannot read every record, decide how you are sampling. Two honest moves:

• Random sample with a seed. Number the records, generate random numbers with a fixed seed, read the first N. The seed matters — it lets a critic reproduce your selection.
• Maximum-variation sample. Pick records that disagree with each other, then add records that fill the gap between them. This is good for exploratory work, weak for final counts.

What you should not do is sample by "the most interesting ones" or "the ones at the top of the search results." Both quietly bake search-engine ranking into your conclusions.

A small confession helps

If you are only reading a subset, write a line in your brief that says so. "Read 60 of 412 filings; selection bias likely over-represents recent, English-language, U.S.-domiciled sources." A reader will trust the rest of the brief more, not less.

Info

Count what you can, qualify what you cannot

Numbers are not the only way to use content as data, but they are the most over-rewarded one. A precise figure from a sloppy source is worse than a rough range with the source spelled out. Prefer:

• Counts of records you actually read
• Date ranges with min and max
• Triangulated estimates with a clear method

If a number is going in your brief, it should be reproducible from your records in three steps by someone who has never seen your work. If it is not, it is rhetoric dressed up as data.

The honest summary of the last few years in open-source work is that AI moved three specific bottlenecks and left most of the rest of the field in place. The chapter names what changed, what did not, and where the new seams are.

What was hard before 2023

Five years ago, an OSINT workflow for any non-English-heavy topic looked like this:

1. Manually scrape or screenshot pages of interest.
2. Read each page in full, in the source language.
3. Manually transcribe the relevant sentence into your notes.
4. Decide what to do with disagreement between sources.
5. Write the brief, by hand.

Steps 1, 2, and 3 are the parts that ate time. A typical investigator could read maybe 30 to 60 pages of dense primary material in a working day, and the cost of crossing a language barrier was a paid translator or a colleague.

What changed

By 2023–2024, three things were suddenly cheap:

• Search and discovery in a wider set of languages. Even imperfect machine translation is enough to know whether a page is worth a full read.
• Summarization that lets you triage a long document and return to the parts that matter.
• Extraction of structured fields — names, dates, dollar amounts, addresses — out of messy pages, with much less custom code.

For a one-person team, that is roughly a 3–5x increase in pages read per day, with better coverage of non-English sources. For larger newsrooms, the multipliers are similar but the absolute volume of material is now the limiting factor instead of the reading speed.

What did not change

The parts of OSINT that the language models did not touch are the parts that were always the point:

• Framing the right question. Still a human skill, possibly more so when the cost of "I just summarized the whole web" is so low.
• Knowing which surface to trust. Still built from experience and reputation, not from a benchmark.
• Reading a document carefully enough to catch the sentence that contradicts the headline. Still requires actually reading it.
• Adjudicating disagreement between sources. Still requires a model of who has the incentive to lie, misremember, or shade.
• Writing a brief a stranger can audit. Still requires the willingness to put your reasoning on the page.

The bottleneck moved up the stack

Before 2023, the hard part was reading enough. After 2023, the hard part is deciding what is worth reading, and being honest about what your summary is leaving out. That is a more interesting problem, and a harder one to outsource.

Tip

The new failure modes

The same capabilities that made AI useful made three new classes of error easier than ever to commit:

• Confident hallucination. A model that summarizes a PDF you do not have in front of you will happily invent quotes. Always re-check the exact wording against the source.
• Citation laundering. A model that "looks up" a fact may be reporting a pattern it learned during training, not a current page. The URL it produces may not exist.
• Bias by training data. If your model was trained on the open web, it has read the same misinformation everyone else has. Its confidence on contested topics is not evidence.

The chapters that follow are organized around these new failure modes. The rest of this guide treats AI as a tool in the loop — sometimes the most useful one, never the only one, and always answerable to the records you actually have.

A language model is the first tool in OSINT that can plausibly do the parts of the work you are bad at — synthesis, translation, draft writing — and is also the first tool that will confidently lie to your face about what it just did. The chapter is about how to keep the first property without inheriting the second.

Three rules that hold up

These are not "best practices" in the soft sense. They are the rules that keep the model from quietly doing something you cannot defend.

1. The model only sees what you give it

When you paste a document into the prompt, you are creating a new mini-corpus that the model reasons over. When you ask a model to "summarize the news about X," the model is reasoning over a vague memory of its training data — which may be months or years out of date, may include misinformation, and may include nothing at all about your specific case.

The first move is almost always the second one. Give the model the source, then ask. If you cannot give the model the source, you are not doing OSINT — you are doing a vibe check.

2. The model summarizes; you decide

A model can compress a 30-page filing into a paragraph, list the named entities in 1,000 tweets, or draft a one-page brief in your voice. None of those are decisions. The model does not know whether the filing is the right one, whether the named entity list is biased, or whether the brief is true.

The judgment stays with you

A useful mental model: the model is a junior analyst with a perfect memory of what you handed them and no idea whether the assignment is real. You are the editor. Their draft is a starting point, not a citation.

Info

3. The model is a compression algorithm, not a source

When a model produces a sentence, that sentence is a compression of patterns in the input (and, if you did not ground the prompt, in the training data). It is not a source. If a sentence is going to appear in your brief, it should be traceable to a record you control, not to a model output.

This rule sounds pedantic until you watch a friend lose two days defending a quote that the model fabricated. Then it sounds like the only rule that matters.

Prompts that help

The following patterns are general enough to apply to most tasks. Adapt them to the model you are using.

Triage a long document

"Read the attached PDF. List the 5 most important claims. For each, quote the exact sentence and the page number. Do not add claims that are not in the document. If a section is unclear, say so."

The "do not add claims" instruction is doing real work. Without it, models will often fill in plausible context that sounds like the document and is not.

Extract a structured table

"From the attached filings, extract one row per company: name, registration number, jurisdiction, date of incorporation, listed officers. Return as a markdown table. If a field is missing, write 'missing' — do not guess."

Translate for triage

"Translate the following 20 pages from Portuguese to English. I am only deciding whether to read the originals. Keep named entities, dates, and numbers exactly as they appear. Do not summarize."

Draft a brief from records you control

"I will paste 12 records from my spreadsheet, in the order they appear. Write a 400-word brief in plain English that supports the following claim. Use only the records I have given you. Where two records disagree, surface the disagreement. End with a one-paragraph 'what would change my mind.'"

Prompts that hurt

The following patterns are common and almost always produce work you cannot defend.

• "Summarize everything you know about X." Produces a confident but unsourced answer.
• "Cite your sources." Produces URLs that look right and may not exist. Always click through.
• "Pretend you are an expert in Y." Increases confidence without increasing accuracy.
• "What does the literature say about Z?" Produces plausible-sounding citations to papers that may not exist or may not say what the model says they say.

Never trust a model to cite itself

If a citation came from a model, treat it as a lead, not a source. Open the cited page. If the page does not say what the model said it said, drop the claim.

Warning

Habits that scale

Three habits make AI-assisted OSINT survive review:

• Always store the prompt. The prompt is part of the record. A future reader — including you — should be able to reproduce the model's output from the prompt and the input.
• Always store the input. The PDF the model read is the evidence. The output is a paraphrase of that evidence.
• Always store your own correction. The places where you changed the model's draft are the analysis. A redline of model → final is often the most useful artifact in the project.

Insight is not volume. A thousand pages of public content, read passively, give you the same understanding of a topic as the first three — sometimes worse, because the thousandth page encourages the illusion of having done the work. The point of this chapter is to give you four reading moves that turn content into insight, regardless of the topic.

Move 1: Read what the source actually says

The first move is the one most people skip. Read the sentence. Then read it again. Then say it back in your own words. If you cannot, you have not read it; you have skimmed it. Skim is fine for triage, fatal for analysis.

A simple test: can you quote the source's strongest sentence from memory, and can you name the part of the argument it supports? If yes, you read it. If no, you skimmed it and you should put it back in the triage pile.

The paraphrase tells you the truth

If you cannot paraphrase a source, you are not yet reading it. The act of paraphrasing forces the move from "I saw the words" to "I understand the claim." Most failed OSINT briefs come from the place between those two states.

Tip

Move 2: Read what the source does not say

Every public document has edges. A press release that lists three initiatives and never mentions a fourth one is a finding. A regulatory filing that discusses every risk in the industry and skips one is a finding. A long interview that never names a specific person or number is a finding.

The negative space is where most of the insight lives, because the absence is often a choice, and choices are usually for a reason.

A practical way to read the gaps:

• Build a list of the topics you would expect a source to cover, given its kind. (A press release about a product launch should cover price, availability, and a spokesperson quote. A regulator's enforcement notice should cover the violation, the remedy, and the date.)
• Note which items on that list the source does cover.
• Note which items on that list the source does not.
• The empty cells are leads. So are the items covered out of order, or with unusual emphasis.

Move 3: Read the claim, not the speaker

It is tempting to evaluate a source by who said it. That instinct is sometimes right — a regulator's notice and a competitor's blog post about the same event are not equally weighted — but it is also a shortcut that hides most of the analysis. The move is to read the claim as a claim, separate from the speaker, and ask:

• What is the strongest version of this claim?
• What evidence, if true, would support it?
• What evidence, if true, would contradict it?
• Which side of that ledger is the source actually contributing to?

Now you can attach the speaker as a weight on the claim, instead of letting the speaker decide the claim for you. A claim from a biased source can still be true, and a claim from a trusted source can still be wrong.

Move 4: Read across many sources for the pattern

The fourth move is the one that turns content into data. Read across the corpus for patterns that no individual source contains:

• Repetition. If a fact appears in independent sources, that is a signal. Track the first time it appears and how it spreads.
• Divergence. If sources disagree, the disagreement is a finding. The shape of the disagreement often tells you more than the answer to it.
• Drift. A claim that softens or hardens over time is a finding. So is a claim that stays exactly the same.
• Omission. A fact that is in the corpus but is missing from the major summaries is a finding. So is a fact that is in every summary but absent from the primary sources.

A pattern is not a fact

Patterns are evidence. They are not, on their own, facts. A claim that ten sources repeat is more likely to be true than a claim that one source makes — but "ten sources repeat" can also mean "ten sources copied from each other," which is a different problem with the same look.

Info

Putting the four moves together

A short, opinionated workflow that uses all four:

1. Triage with a model. Skim the long list, surface the 10–20 records most likely to matter.
2. Read those 10–20 carefully, in full. Paraphrase each in one sentence.
3. Build the empty-cell grid. Note what each source does and does not cover.
4. Pull the claim, not the speaker, out of each source. Score the strongest version.
5. Look across the corpus for repetition, divergence, drift, and omission.
6. Write the brief from the patterns, with the records as evidence.

This is the loop the next two chapters make more rigorous — first with cross-referencing, then with verification.

A single source, however credible, is still a single source. A claim that survives three independent sources is much harder to dismiss. A claim that survives ten independent sources is essentially a fact, unless the ten sources are all copying from each other — which is its own kind of finding. This chapter is about how to do that comparison honestly.

What "independent" means in practice

Two sources are independent when one of them did not get the claim from the other. The bar is higher than it sounds. Sources that look independent often are not:

The way to test independence is to ask: if I remove the claim from source A, would source B still have it? If the answer is "I do not know" or "probably not," the two are not as independent as they look.

The independence test is the most under-used move in OSINT

Most "three sources confirm" claims are really "one source confirmed by two outlets that copied it." A brief that names the chain of dependence is rarer, and much more useful.

Warning

Triangulation, in three steps

A practical triangulation looks like this:

Step 1 — Name the claim precisely

Not "the company is doing well," but "the company reported $X in revenue for FY2024, a Y% increase over FY2023, in its audited annual filing dated D." The precise version is the only one you can confirm.

Step 2 — Find sources that could, in principle, disagree

If every source you can find agrees, you may be looking at a small pond. The useful sources are the ones that might have a reason to disagree. For a corporate claim, that includes:

• The audited filing
• Investor presentations
• Press coverage from outlets not given a press release
• Employee or contractor accounts on public forums
• Industry analysts with a different thesis
• Regulators, competitors, or customers with skin in the game

Step 3 — Write down what each source actually does to the claim

A useful habit: for each source, write one of four labels.

• Supports — adds evidence in favor of the precise claim
• Contradicts — adds evidence against it
• Refines — supports the claim with a different number, scope, or boundary
• Silent — does not speak to it, in which case the source is not evidence at all

A pile of "silents" is not confirmation. It is just silence.

Weighting without flinching

After you have a list of labels, you have to weight them. Weighting is the part that gets criticized, because it requires you to be explicit about what you are doing. A defensible scheme:

• Primary, audited, public-interest source (regulator, court, exchange) — strongest single weight
• Primary, unaudited, in-scope (company filing, statement) — strong on facts the source has no reason to distort
• Reputable secondary (long-form outlet with named reporter) — useful, especially when they describe how they got the fact
• Anonymous or self-published primary (a person's post, a forum) — useful as a lead, weak as a sole source
• AI summary or model output — never a weight; it is a triage layer, not a source

Two weights at the top outrank a dozen at the bottom. Three weights at the top, on independent angles, outrank any number lower down.

Show your weights

In the brief, write the weighting in plain language. "Regulator filing (primary, audited) and one independent newsroom investigation agree on the date; company press release silent." That sentence is what makes the rest of the brief auditable.

Tip

What to do when sources disagree

Sometimes the triangulation will not resolve. Honest moves, in roughly increasing cost:

• Disclose the disagreement in the brief. "Sources A and B report X; source C reports not-X, and gives a different definition of the underlying metric." This is usually enough.
• Hunt for the upstream source. If A and B both cite C, find C and see what it actually said. Often the chain of copying has stretched a claim past its meaning.
• Apply the incentives test. Who benefits if the disputed claim is true? Who benefits if it is false? That test does not resolve the fact, but it often tells you where to put your confidence.
• Wait. Some disagreements do not resolve until a primary source is published, an audit is released, or a court records a finding. A short brief that says "as of writing, the two filings disagree" is more honest than a long brief that pretends they do not.

What you should not do: pick the source you like, drop the others, and write a confident paragraph. That paragraph is what gets a research project in trouble.

A few years ago, verifying public content mostly meant "is this real?" Today, that question has split into two: "is this real?" and "is this what it claims to be?" A real video of a building may have been filmed last week or ten years ago. A real quote may belong to a different speaker. A real document may be a forgery of a real document. The chapter is a short field guide to the moves that still work, and the moves that no longer do.

Start with the file, not the claim

The fastest way to lose two days on a fake is to start with the claim ("is this photo from the right city?") instead of the file ("is this photo a real photo?"). Verification always runs from the bottom up.

A useful order:

1. Is the file what it says it is? Format, metadata, consistency.
2. Is the source who they say they are? Provenance, channel, history.
3. Is the content what it claims to be? Place, time, people, objects.
4. Is the claim supported? Frame, caption, and the rest of the corpus.

Skipping step 1 to ask step 4 is how people get fooled by old, mislabeled, or AI-generated images that nonetheless "look like" the topic.

Image checks that are still useful

Metadata

A JPEG or PNG can carry EXIF data: camera model, capture time, sometimes GPS. EXIF is easy to strip and easy to fake. Its absence tells you nothing. Its presence is a small positive signal.

The order of edits is in the file

A re-saved JPEG has different compression artifacts from the original. A screenshot has none. The tool FotoForensics runs error-level analysis on the file and shows you the spots where the file has been re-saved, which often correspond to edits.

Tip

Reverse image search

TinEye, Google Images, and Yandex Images all run reverse image search. The point is not just to find duplicates — it is to find the earliest version. The earliest version often tells you:

• Where the image was first published
• Whether the original was cropped, mirrored, or color-graded
• Whether the original caption is different

Visual cross-reference

For a geolocation, the move is to look for the architectural or environmental features in the image — building shapes, sign fonts, mountain ridges, vegetation, the position of the sun — and match them to maps and street view. This is the slowest step and the most reliable.

A practical way: pin the image to a side-by-side with a candidate map view. Look for the silhouette, the road markings, the language on the signs, the roof style. A match on at least three independent features is usually enough.

Video checks

A short, viral video is the easiest thing in the world to fake, and the hardest thing in the world to verify. Three moves help:

• Split the clip into frames. InVID / WeVerify or a free ffmpeg command will give you the keyframes. Many of the tells in synthetic video only show up at full resolution on a single frame.
• Check the audio and the visual separately. A deepfake of a person's face often has perfectly clean audio, or audio that does not match the lip movements. The mismatch is the tell.
• Look for the edges. Synthetic video tends to glitch on the boundary between the generated face and the real background. Earrings, glasses frames, hair strands, and the boundary of the jaw are the usual suspects.

A clean second-generation copy is not a clean original

A video that has been re-encoded by a platform has lost most of the metadata. That does not mean the video is fake. It means you have to verify from the pixels, not the file.

Warning

Document checks

For PDFs and images of documents — the format most leaks arrive in — three moves help:

• Look at the typeface, the kerning, the line breaks. Real documents from real software have specific artifacts. A "leaked" PDF whose lines wrap oddly or whose font is unusual for the alleged author is worth a closer look.
• Check the metadata of the file itself. PDF metadata includes the software that created it, the author, and the modification history. None of this is conclusive — it can all be edited — but it is a check worth running.
• Search the document for a sentence. Paste a distinctive sentence from the document into a search engine. If the same sentence appears in another document, you have a lead on provenance.

Quotes and speech

For a quote that may or may not have been said:

• Find the source of the quote. Where was it first reported? Was the original a transcript, a recording, a paraphrase? A quote that has been re-quoted a hundred times often has a long chain of small distortions.
• Compare to the original. If the original is a transcript, compare the exact wording. If the original is a recording, listen for the inflection. Quotes in writing are not the same thing as what was said out loud.
• Treat the audio as a separate object. A clean audio recording of a known speaker is strong evidence. A synthesized audio clip of a known speaker saying something they never said is the new failure mode. When in doubt, treat the audio as the claim, not the source.

When the verification does not resolve

Sometimes you will not be able to confirm or kill a piece of content. That is a real and common outcome. The move is to write the result honestly:

"The image was uploaded to account X on date D. Reverse image search returns earlier versions from at least three other sites, none of which is the claimed source. We cannot confirm the original capture time and have not been able to geolocate the building."

That sentence is more useful than a confident "looks legit." A future researcher, including you, can pick up the thread.

A toolchain is the answer to "if I closed this laptop and a stranger opened it, could they redo the work?" If the answer is no, the toolchain is too clever. The goal of this chapter is the minimum setup that makes a one-person OSINT project auditable, repeatable, and recoverable when something goes wrong.

The five things that earn their place

You can do meaningful OSINT with five components. None of them are exotic, and most of them are free.

1. A research browser

A second browser profile dedicated to your project. Not your normal browser, with your saved logins, your history, your cookies. A clean profile with the extensions you need and nothing else.

Useful extensions:

• uBlock Origin — fewer trackers, fewer pages that get in your way
• Wayback Machine — one click to archive a page and one click to see if a snapshot exists
• InVID / WeVerify — for the video and image checks described in the verification chapter
• SingleFile — saves a page as a single HTML file you can re-read offline

2. A note system with two surfaces

The cheapest workable setup is a folder of plain Markdown files. The point of Markdown is not the syntax — it is that the file is portable, durable, and you can search across it with grep if you have to.

Use two files:

• A running notes file. One section per day, in chronological order. Date headings. Stream of consciousness. This is the file you write in.
• A records file. One row per source, in tabular form. This is the file you cite from.

Every record in the records file should be linked from the notes file, and every claim in the notes file should be linked back to a record. The two-way link is what makes the project auditable.

3. A capture system that survives deletion

The single most common OSINT failure is discovering that a key page has been deleted, edited, or moved. The fix is to capture a copy that you control, in two places:

• A local copy — saved as PDF, single-page HTML, or screenshot
• An archived copy — saved to the Wayback Machine via the browser extension

The local copy lets you re-read and re-quote offline. The archived copy lets you prove the page existed at a specific time, even if the original is gone.

4. A model you can talk to

A language model you can paste documents into, with a context window large enough to hold the records for one project. The point is not the size of the model — it is the discipline of always pasting the source, not asking the model to "look it up."

A small open-weight model running locally (via Ollama) has one big advantage: nothing leaves your machine. That matters when the records are sensitive. For open material, a hosted model is fine, as long as the prompts and inputs are part of your records.

5. A reproducible export

A small, deliberate step: at the end of each project, export the notes, the records, the captures, and the prompt log to a single folder. Drop the folder in cloud storage, on an external drive, or in a long-term archive. A year from now, when someone asks "where did you get that?", the answer is a folder, not a memory.

A few scripts that pay for themselves

The following are short enough to fit in a scratch file and useful enough to keep around. They are written in Python because Python is the most common second language in OSINT shops, but equivalents exist in JavaScript, Go, or whatever you prefer.

Pull every link from a page

import requests, re, sys
url = sys.argv[1]
html = requests.get(url, timeout=20).text
for href in re.findall(r'href="([^"]+)"', html):
    print(href)

Check whether a list of URLs is still live

import concurrent.futures, requests
urls = open("urls.txt").read().splitlines()

def head(u):
    try:
        r = requests.head(u, timeout=10, allow_redirects=True)
        return u, r.status_code
    except Exception as e:
        return u, f"err:{e.__class__.__name__}"

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as ex:
    for u, code in ex.map(head, urls):
        print(code, u)

Archive a URL to the Wayback Machine

import requests
requests.post(
    "https://web.archive.org/save/",
    headers={"User-Agent": "your-project-name/1.0"},
    data={"url": "https://example.com/page"},
)

The script is the artifact, not the output

When you write a script that helps, keep the script. The next project will want a slightly different version of the same idea, and you will not remember the details in three months. A folder of small scripts is more useful than any single tool.

Tip

What to leave out

A toolchain earns its place. A few things to leave out of your default setup:

• Scrapers that require paid APIs. Most projects can be done with requests and a polite sleep between calls.
• Closed-source "OSINT platforms" with no documentation. If you cannot describe what the tool actually did, you cannot describe what your project actually found.
• Anything that requires you to log in with a personal account. Your project's audit trail should not depend on a vendor staying in business.
• Anything that needs a GPU you do not have. Local models are nice; they are not a requirement for a one-person project.

A small, well-understood toolchain is more useful than a large one you half-remember.

Most of the lines in this chapter are not the legal minimum. They are the practical ethics of working with public material in a way that produces insight you can defend and that does not hurt people who did not sign up to be in your project. The legal lines vary by country; the ethical ones travel better.

The legal lines

The legal floor for OSINT is mostly familiar. The areas where one-person projects most often trip:

• Computer-misuse laws. In many jurisdictions, accessing a system you are not authorized to access is illegal even if no password is set. The relevant test is often "would a reasonable person believe the system was public," and the answer is rarely "yes" for a personal inbox, a private server, or a closed forum you were emailed an invite to.
• Copyright. Open does not mean free to republish. A press release is open; printing a 1,000-word excerpt in a brief is usually fine under fair use; printing the full text in a post is not. A photograph is open to view; republishing it without permission is often not.
• Data-protection law. In the EU, the UK, and a growing list of countries, "personal data" of identifiable people is regulated even if it is public. A spreadsheet of every social post by a private individual may be lawful to read and unlawful to publish.
• Defamation. A false statement of fact that harms a person's reputation is treated differently from an opinion. A short, hedged, sourced statement is usually safer than a long, confident one.

A reasonable reader test

The simplest legal test is also the most useful: would a reasonable reader, given the same evidence, come to the same conclusion? If the answer is "no, the reader would have to take my word for it," the conclusion is not yet supported.

Warning

The privacy lines

The most common ethical mistake in OSINT is to treat a public account as a public person. They are not the same. Some useful rules of thumb:

• Stay on the platform. A public Instagram account is a license to look at the public Instagram account. It is not a license to download every post, run face recognition on them, and publish the result as a dataset.
• Aggregate, do not identify. A finding that "ten accounts in this network share a common posting pattern" is a finding about a network. A finding that lists the ten accounts is a different kind of finding, with a different cost, and should be justified in proportion to the public interest.
• Minimize. Pull the data you need for the question you are answering, not the data that might be interesting later. The data you did not collect cannot be leaked.
• Be kind to sources. People who appear in your records are not characters in your project. If a source later asks for a record to be corrected, the move is to correct it, not to argue.

The doxxing line

Doxxing — publishing a person's home address, workplace, daily route, or other private details — is not OSINT. It is the misuse of OSINT. The line is usually drawn at:

• Information the person has clearly chosen to keep private, even if it is technically public somewhere
• Information whose publication has no clear connection to a public interest
• Information whose publication puts the person at risk of physical, financial, or reputational harm beyond the legitimate cost of being criticized

A useful test: would you publish this information about yourself, in this format, on this platform, with this audience? If no, do not publish it about someone else.

Bias in the corpus

The corpus you collect is shaped by the surfaces you can reach. Most one-person projects are biased in three predictable ways:

• Language bias. English (and, to a lesser extent, Spanish, French, Mandarin, and Russian) is over-represented. Sources in other languages are missed.
• Recency bias. Recent pages rise to the top of search results. Older primary material disappears unless you go looking for it.
• Survivor bias. Deleted posts, retracted papers, bankrupt companies, and dead domains are not in your search results. The fact that you cannot find them is not evidence that they did not exist.

The move is not to pretend the bias is gone. The move is to name the bias in the brief, and to design the corpus to push against it in the places that matter for the question.

The brief is the place to say it

One short paragraph in the methods section, naming the three biggest biases of the corpus and what you did about them, is more useful to a reader than 20% more sources. The reader cannot calibrate a brief they cannot audit.

Tip

Bias in the reader

The harder bias is the one the OSINT practitioner brings to the corpus. Confirmation bias, motivated reasoning, and the sunk-cost fallacy are not exotic — they are the default state of a person who has spent two weeks on a project. Some moves that help:

• Pre-register a falsification. Before you start, write down the specific kind of evidence that would change your mind. When you find it, stop and write the finding before you do anything else.
• Show the negative evidence. A brief that lists the records that did not support the conclusion is harder to dismiss than a brief that lists only the records that did.
• Have a hostile reader. Before publishing, give the brief to someone whose job is to find the holes. The first round of "but what about…" is the most useful feedback you will get.
• Time-box the project. A project that has been running for a year is rarely the place where the next clean insight lives. A short, scoped project is more often a publishable unit.

A short ethics checklist

Before publishing, run this list:

• Every claim is linked to a public record the reader can verify.
• Every person named is named for a reason that survives the "would I publish this about myself" test.
• The corpus biases are named in the methods section.
• The negative evidence is on the page, not in a footnote.
• The brief says what would change the conclusion.
• The legal review for your jurisdiction is on file.

A "no" on any of these is not a reason not to publish. It is a reason to look at the answer more carefully.