Position Overview:We are seeking a motivated and experienced Web Scraping Developer with 2+ years of hands-on experience to join our team. The ideal candidate will possess strong technical skills in web scraping, data processing, and performance optimization, with expertise in Python, popular scraping frameworks, and large dataset management. You will play a critical role in extracting and processing data from a variety of websites while ensuring compliance with legal and ethical guidelines. Key Responsibilities:Develop and maintain scalable and optimized Python-based web scraping scripts.Use web scraping frameworks (Scrapy, Beautiful Soup, Selenium, etc.) to extract data from static and dynamic websites.Implement solutions for handling dynamic content using headless browsers like Playwright or Puppeteer.Extract and process data efficiently from complex HTML, CSS, JavaScript, and XPath structures.Work with large datasets, using tools such as Pandas for data manipulation, cleaning, and processing.Ensure proper handling of data export in formats like CSV, JSON, and direct database integration.Consume and integrate REST APIs, manage API rate limits, and handle HTTP protocols, cookies, and sessions.Manage data storage using relational databases like MySQL, optimising queries and indexing for large datasets.Troubleshoot and bypass common anti-scraping techniques such as CAPTCHAs, IP blocking, and user-agent tracking.Use tools and techniques like rotating proxies, headless browsers, and CAPTCHA-solving services to mitigate blocking.Collaborate with teams using version control systems like Git, perform code reviews, and contribute to collaborative workflows.Optimize scraping scripts for performance, including parallel processing or distributed scraping with tools like Celery, Redis, or AWS Lambda.Deploy scraping solutions using tools like Docker, AWS, or Google Cloud, and automate scraping tasks with schedulers (e.g., Cron).Implement robust error-handling mechanisms and monitor scraping jobs with logging frameworks.Stay updated with web scraping trends and ensure that projects comply with web scraping ethics, copyright laws, and website terms of service. Required Qualifications:2+ years of hands-on experience in web scraping using Python.Strong proficiency in Scrapy, Beautiful Soup, Selenium, or similar scraping frameworks.Experience with headless browsers like Playwright or Puppeteer for handling complex websites.In-depth understanding of HTML, CSS, XPath, and JavaScript for dynamic content interaction.Proficiency in data handling with Pandas and experience in exporting data in multiple formats (CSV, JSON, databases).Strong knowledge of REST APIs and working with web protocols like HTTP, cookies, and session management.Experience in managing MySQL databases, query optimization, and indexing for large-scale data.Familiarity with anti-scraping techniques and proficiency in bypassing measures like CAPTCHAs and IP blocks.Experience with version control tools like Git and familiarity with collaborative workflows and code review processes.Hands-on experience in performance optimization, parallel processing, and distributed scraping.Knowledge of deploying scraping solutions using Docker, AWS, or Google Cloud.Strong problem-solving skills, including error-handling and debugging using logging frameworks.Awareness of web scraping ethics, copyright laws, and legal compliance with website terms of service. Preferred Qualifications:Familiarity with cloud-based solutions like AWS Lambda, Google Cloud, or Azure for distributed scraping.Experience with workflow automation tools and Cron jobs.Basic knowledge of frontend development (HTML, CSS, JavaScript) is a plus.
