How to Collect Social Media Posts as Data: A 5-Step Pipeline

How to Collect Social Media Posts as Data

Collecting social posts is not the same as scraping them. Scraping is a technical action; collecting is a research action that includes planning, ethics, and a defensible record. The five-step pipeline below is the one that turns a viral feed into a dataset a stranger can audit.

Stay on the platform, do not bypass technical controls

The legal floor for social media research varies by country and platform, but the working rule is universal: stay on the platform, do not bypass access controls, do not aggregate in a way that re-identifies private individuals, and do not publish the dataset in a way that puts people at risk. The platform's terms of service are a contract, not a law — but breaking them can lose your access, and in some jurisdictions can expose you to liability.

Warning

Step 1: Define the surface

A surface is a single source family. "Twitter" is not a surface — it is a platform. "Public tweets from accounts in country X between dates D1 and D2 containing keyword Y" is a surface.

Defining the surface means writing down:

• The platform. Twitter / X, Reddit, TikTok, YouTube, a forum, a comment section, a public Telegram channel.
• The account type. Verified, unverified, organization, individual, anonymous.
• The visibility. Public, semi-public (followers-only), private. Only public is in scope for most OSINT work.
• The time window. Specific dates, or "all available." The window matters because the corpus bias of "all available" is usually "whatever the platform's API decided to return."
• The query. Keywords, hashtags, user lists, geographic filters.

A surface that is not defined is a surface that will quietly drift as the researcher notices interesting posts and adds them. The drift is a bias, and the bias is a finding the reader should know about.

Step 2: Freeze the snapshot

The moment you decide a post matters for your research, capture a copy in two places:

• A local copy. Screenshot, single-page HTML, or a platform's own export. The local copy is what you cite from.
• An archived copy. Saved to the Wayback Machine or archive.today. The archived copy is what survives deletion.

The freeze matters because the original post can be deleted, edited, or made private at any time. A dataset that depends on the live platform is a dataset that breaks the moment the platform changes. A dataset that depends on archived copies is durable.

Save the URL, the timestamp, and the platform's metadata

Each record should include the URL of the post, the timestamp you captured it, the author handle, the author's display name, the post text, and any platform metadata (post ID, view count, like count at capture time). The platform metadata is itself data — a post with 1,200 likes at capture and 1,200 likes today is different from a post with 12 likes at capture and 1,200 likes today.

Tip

Step 3: Structure the records

Raw collection is a mess. Turn each post into a record with the same shape, with at minimum:

• id — a stable identifier you control (e.g., post-001)
• author_handle — the @-handle
• author_display_name — the name displayed on the post
• platform — Twitter, Reddit, etc.
• post_url — the canonical URL
• post_id — the platform's own ID
• text — the post text
• created_at_platform — when the platform says the post was created
• captured_at — when you captured it
• archive_url — the Wayback Machine or archive.today URL
• local_path — where your local copy is stored
• engagement_at_capture — likes, shares, comments, views (whatever the platform exposes)
• language — the language of the post
• media_urls — any attached media
• notes — your own one-line observation

The point is not bureaucracy. The point is that a record you cannot find next month is the same as no record at all.

Step 4: Document the limits

Before you start analyzing, write a short paragraph on the limits of the corpus. Typical limits:

• API limits. Most platforms return only the most recent N posts, or the most recent N posts matching a query, even when more exist. The corpus is whatever the API gave you.
• Search bias. Most platform search engines rank by recency and engagement, not by relevance to your query. A search for keyword Y returns the recent and engaged posts first.
• Language bias. Most platform search engines do not translate or cross-language. A search in English misses posts in other languages.
• Deletion bias. Posts that were deleted before you started collecting are missing. A corpus of public posts is a corpus of posts that survived until you looked.
• Account bias. Posts from suspended, banned, or deleted accounts are not in the corpus. So are posts from accounts you did not know to look at.

The brief is the place to say it. "We collected 1,200 public posts matching the query between dates D1 and D2. The platform's search engine returned the most recent 1,200; we do not have visibility into posts that were deleted before collection. Search bias likely over-represents recent, high-engagement posts." A reader will trust the rest of the brief more, not less.

Step 5: Archive the originals at the end

The final step, after analysis but before publication, is to make sure the corpus is durable:

• Local copy. All records and all local captures in a single project folder.
• Archive copy. A snapshot of the public-facing dataset in a long-term archive (Internet Archive, Zenodo, your own durable storage).
• Documentation. The query, the date, the platform, the limits, the redline from raw to processed.

A project that does not do this is a project that has to be redone when the platform changes, the laptop dies, or the author moves on.

A small script for the freeze

A useful move: the moment you decide a post matters, run a script that captures the local copy and the archive copy in one shot. Something like:

import requests, json, sys
post_url = sys.argv[1]

# 1. Local copy
html = requests.get(post_url, timeout=20).text
with open(f"captures/{post_url.split('/')[-1]}.html", "w") as f:
    f.write(html)

# 2. Archive copy
requests.post(
    "https://web.archive.org/save/",
    headers={"User-Agent": "your-project-name/1.0"},
    data={"url": post_url},
)

The script is small. The discipline is the larger thing. Run it every time, and your dataset survives.

The move that catches the most misses

A single freeze is good. A freeze plus a record of the platform's metadata at capture time is much better. The post's text is durable; the post's engagement is not. A claim that "this post went viral" should be backed by the engagement numbers at capture, not by the numbers today.

The most expensive research mistakes happen when only the text is preserved. A post that is unremarkable at capture and viral at publication is a different finding than a post that was viral at capture and unremarkable at publication. The capture-time engagement is part of the record.