NanmiCoder/MediaCrawler

If you need a corpus of Chinese social-media posts and comments for study, MediaCrawler is one of the few projects that covers seven platforms in one codebase. It is a scraper, not a turnkey dataset, and the open-source edition deliberately stops short of some operational features.

The problem

A researcher who wants to study discussion around a topic on Xiaohongshu, Douyin, Bilibili or Zhihu typically has to write a separate scraper per platform, each with its own login handling, request signing and pagination quirks. The front half of that work — figuring out how to authenticate, resolve a keyword into results, and page through nested comment threads — is repeated every time. MediaCrawler narrows that to choosing a platform and a crawl type: keyword search, specific post IDs, a creator's homepage, and second-level comments. The output becomes a local dataset of posts and comments that can be analyzed or turned into a word cloud.

What's different

The distinctive choice is how it obtains data: rather than reimplementing each platform's signature algorithm, it drives a real browser through Playwright. The default mode connects over CDP to an existing Chrome instance, which means a cached login session can be reused across runs. That approach trades speed and resource efficiency for the practical benefit that a working login state is enough to get going, and it removes the need to install a browser driver in CDP mode. Sessions, cookies and IP proxy pools are supported per the feature table.

The cost is the inverse side of the same design. Running a browser per crawl is heavier than plain HTTP requests — the project's paid Pro edition explicitly advertises dropping the Playwright dependency as an advantage. The feature table also shows the same matrix for all seven platforms, but the honest read is that the scraping surface is uniform while the underlying page structures are not; expect platform-specific quirks in login flows and in how deeply comments nest. Caching login state is a convenience, not a guarantee that a session stays valid, so repeated runs may still need re-authentication.

Who should try it

Good fit: a student or researcher assembling a one-off or recurring corpus from Chinese platforms, who is comfortable setting up Python and a browser and wants to avoid writing seven scrapers. The WebUI and keyword-search path lower the bar for people who mainly want data rather than code. Poor fit: someone who needs unattended, long-running large-scale collection across many accounts. The README places resume-from-breakpoint crawling, multi-account plus proxy-pool operation, and full Linux support in the Pro edition, so those who need interruption-tolerant pipelines should evaluate that separately rather than assume the open-source edition covers it.

Requirements and limitations

Setup requires Python tooling (uv is recommended), Node.js 16 or newer, and either a running Chrome instance for CDP mode or Playwright's browser driver for the standard mode. The install commands are documented in the project README, and uv itself comes from the official uv installation guide. A live demo page exists for the WebUI.

Ongoing use carries two burdens the README does not resolve. First, scraping social platforms is a moving target: when a platform changes its login or page structure, the tool can stop working until it is updated, so budget time for breakage. Second, and more important, the README's stated purpose restriction forbids illegal use and commercial uses outside study and research; users are expected to use it only for personal learning and technical research. That is a licensing-and-ethics boundary to check before adopting it in any organizational project.

HotRepo's take

HotRepo's tracked history begins on 2026-10-24, and the only daily-board observation is 2026-10-25, when the project stood at 68,460 stars with a gain of 408 that day. That single snapshot supports the scale of attention but not a trend; persistence cannot yet be judged. The decisive check before adopting is the project's usage restriction and whether you need the Pro-only continuity features — not the star count.