[Remote] Member of Engineering (Pre-training / Data Acquisition)
Note: The job is a remote job and is open to candidates in USA. Poolside is a company focused on building a world where AI drives economically valuable work and scientific progress. The role involves working with the pre-training data team to acquire high-quality pre-training data for frontier models, ensuring alignment with training needs and building systems for efficient data acquisition.
Responsibilities
- Design, build, and operate a large-scale web crawler responsible for acquiring all openly accessible data on the internet
- Develop specialized deep crawlers targeting high-value sources to improve recall and coverage
- In collaboration with data researchers, own a long-term road map for data acquisition
- Build observability, monitoring, and debugging tooling to ensure reliability and transparency across crawl infrastructure
- Collaborate with pre-training, post-training, and evaluations teams to align data acquisition priorities with model training needs
- Build high-throughput ingestion pipelines for rapidly onboarding partner data and evaluating it for quality
Skills
- Strong distributed systems background with proven experience building and operating large-scale infrastructure — data pipelines, web crawlers, or similar
- Proficiency in Python, and comfortable optimizing performance and debugging complex systems under production conditions
- Hands-on experience with web crawling or large-scale data extraction: understanding of HTTP protocols, distributed job queues, and data parsing at scale
- Familiarity with cloud platforms (AWS) and container orchestration (Kubernetes, Docker) for deploying and managing high-throughput workloads
- Awareness of the non-technical dimensions of internet-scale crawling: data privacy, robots.txt adherence, and responsible crawl practices
- Prior experience pre-training LLMs
- Experience in building trillion-scale SOTA pre-training datasets
- Experience translating research to production at scale
Benefits
- Fully remote work & flexible hours
- 37 days/year of vacation & holidays
- 16 weeks of flexible, full-pay parental leave
- Health insurance allowance for you & dependents
- Company-provided equipment
- Well-being, always-be-learning & home office allowances
- Frequent team get togethers
- Diverse & inclusive people-first culture
Company Overview
Company H1B Sponsorship