AI/ML Web Scraping Specialist

AI/ML Web Scraping Specialist

Key Responsibilities

• Design and implement advanced web scraping tools, scripts, and pipelines using Python (BeautifulSoup, Scrapy, Selenium, Playwright, etc.).
• Build robust scrapers for social media platforms like Instagram, TikTok, X (Twitter), Facebook, and LinkedIn, bypassing rate limits and anti-bot mechanisms (Cloudflare, reCAPTCHA, etc.).
• Leverage APIs (when available) and reverse-engineer web/mobile requests to extract structured data.
• Develop and train ML models for tasks such as content categorization, influencer classification, sentiment analysis, engagement prediction, etc.
• Automate scraping workflows, schedule jobs (using Airflow/Cron), and store data in NoSQL or relational databases.
• Maintain and optimize scraping performance, and handle edge cases or UI changes in target platforms.
• Work with large-scale data pipelines, and ensure clean, deduplicated, and enriched datasets.
• Collaborate with product, marketing, and data science teams to provide actionable insights from social media data.

Required Skills and Qualifications

• Strong programming skills in 

  Python

   with proven experience using 

  Scrapy, Selenium, Playwright, or Puppeteer

  .
• Deep knowledge of HTTP, HTML DOM traversal, JavaScript rendering, proxies, user agents, and browser automation.
• Solid understanding of 

  Instagram’s data structures

  , public endpoints, GraphQL queries, and security challenges.
• Familiarity with 

  anti-bot bypass techniques

  : rotating proxies, CAPTCHA solving (2Captcha, AntiCaptcha), session management.
• Hands-on experience in training and deploying 

  ML models

   (NLP, classification, clustering) using 

  scikit-learn, TensorFlow, or PyTorch

  .
• Experience with 

  MongoDB, PostgreSQL, or Elasticsearch

   for data storage and retrieval.
• Good understanding of data privacy, legal considerations, and ethical scraping practices.

Preferred Skills

• Experience with 

  cloud platforms (AWS, GCP, Azure)

   and containerization (Docker/Kubernetes).
• Knowledge of 

  Instagram Business APIs

  , Facebook Graph API, and TikTok’s unofficial endpoints.
• Prior work on influencer discovery, brand monitoring, or social listening tools.
• Experience in building data dashboards using tools like 

  Streamlit, Power BI, or Tableau

  .
• Contributions to open-source scraping libraries or ML projects.

Tools & Technologies You Might Use

• Python, Scrapy, Selenium, Playwright, Puppeteer
• Pandas, NumPy, scikit-learn, OpenAI APIs
• PostgreSQL, MongoDB, Redis, Elasticsearch
• AWS Lambda, EC2, S3, Cloud Functions
• Git, Docker, CI/CD pipelines