Senior Web Scraping Engineer

You will be responsible for building and maintaining scrapers for 24-30 websites (initially starting with 6), handling authenticated sites, sessions, cookies, tokens, etc. Additionally, you will work with search endpoints and dynamic content, architect the data ingestion pipeline, implement caching, retries, and queues, and ensure the stability and resilience of the scrapers even as websites evolve. Collaborating closely with the backend team to define and utilize APIs, you will also manage scaling, rate limits, and performance. Qualifications Required: - 5 - 7+ years of experience in building production-grade web scrapers - Strong proficiency in API-based scraping and browser automation, particularly with Playwright/Puppeteer - Familiarity with proxies/IP rotation and basic anti-bot handling - Comfortable with authentication flows involving cookies, sessions, and tokens - Ability to design pipelines for data ingestion, cleaning, storage, and caching - Capability to work independently and take ownership of the scraper system end-to-end About the Company: The company is focused on building a product that heavily relies on collecting structured data from various known websites. They are seeking an experienced individual who can take charge of this aspect of the system, from creating scrapers to ensuring their scalability and reliability. This senior role requires dealing with real-world challenges of web scraping and the ability to work autonomously. Additional Details: The company values reliability and predictability in scrapers, emphasizes system-oriented thinking over quick fixes, encourages remote work with self-management, and promotes clear communication with minimal bureaucracy. Compensation: Base salary ranges from 3,00,000 to 4,00,000 per month (equivalent to 36 - 48 LPA). Equity/options are available for those willing to accept a lower cash component.,