Research Engineer - Web Crawlers

Overview

Research Engineer (Web Crawlers) to build and own large-scale web crawling systems for frontier AI models.

What you'll do

• Build and operate large-scale, distributed web crawlers for discovery, fetching, and extraction.
• Solve crawling problems including messy-HTML content extraction, deduplication, and freshness/recrawl strategies.
• Design targeted crawling pipelines for high-value sources such as audio, video, and multilingual content.
• Create tooling and infrastructure to let researchers request, monitor, and explore newly crawled data.
• Handle politeness and rate-limit requirements during crawling operations.

What you'll need

• Hands-on experience building and scaling web crawlers or scraping systems for ML training data.
• Strong engineering skills in distributed systems at scale (e.g., Kubernetes, queue-based architectures, or pipelines).
• Ability to autonomously evaluate quality, coverage, and compliance of crawled data.
• Capacity to build tooling to measure data quality, coverage, and compliance.
• No formal certifications or degrees required.

Details

• Work mode: Remote; can be executed globally.
• Optional office locations: London, New York, San Francisco, and Warsaw.