AI Python Web Scrapper

Python Web Scraping Engineer

Key Responsibilities:

• Design and build scalable scraping scripts using

  Python (Scrapy, BeautifulSoup, Selenium, Playwright)

• Extract structured and unstructured data from eCommerce, fashion, marketplace, and social media websites
• Handle dynamic content, lazy loading, and login-based content scraping
• Process and clean scraped data for AI models and data analytics pipelines
• Collaborate with AI/ML engineers to deliver enriched datasets for training and automation
• Bypass anti-scraping techniques using

  rotating proxies, headless browsers, CAPTCHAs solvers

  , etc.
• Implement scraping best practices to maintain anonymity and minimize IP bans
• Store scraped data in

  JSON, CSV, MySQL, MongoDB

  , or push it to APIs
• Schedule scraping tasks using

  cron jobs

  ,

  Airflow

  , or similar tools
• Monitor data quality and update scripts when site structures change
• Maintain scraping logs and update stakeholders with progress reports

Required Skills & Qualifications:

• 3–4 years of experience in

  web scraping with Python

• Proficiency in

  Scrapy, Selenium, Playwright, BeautifulSoup

• Experience with

  CAPTCHA-solving tools

  ,

  proxy rotation

  , and

  IP management

• Familiarity with REST APIs, JSON, XML
• Experience storing data in

  databases

  (MongoDB, PostgreSQL, MySQL, etc.)
• Knowledge of

  version control (Git)

  and basic DevOps is a plus

Preferred (Nice to Have):

• Experience in

  AI/ML projects

  where scraped data supports model training
• Exposure to

  cloud environments

  (AWS, GCP, Azure)
• Prior experience scraping at scale or working with large datasets
• Familiarity with

  text/image data extraction

  (OCR, NLP preprocessing)