TOSS Solutions [ Training | Outsourcing | Sales | Service ]

Job Overview: We are looking for a highly skilled Python Developer with strong experience in building both web and mobile app crawlers. The ideal candidate should have in-depth knowledge of Scrapy, Selenium, REST APIs, proxy rotation, and techniques to bypass anti-scraping mechanisms. You'll be responsible for building scalable, stealthy crawlers that extract data from websites and mobile apps (via APIs or reverse engineering traffic). Responsibilities: Develop and maintain web crawlers using Scrapy, Selenium, and custom spiders. Build and manage mobile app crawlers by intercepting and decoding API requests, using tools like mitmproxy, Charles Proxy, or Burp Suite. Handle proxy rotation, user-agent spoofing, cookie/session management, and headless browsers. Monitor and adapt to changing structures of target websites and apps to maintain scraping accuracy. Optimize performance, manage rate limits, and avoid detection (CAPTCHA, IP blocks). Structure and store extracted data in JSON, CSV, or databases. Maintain logs, error tracking, and reprocessing pipelines for failed jobs. Required Skills: Strong expertise in Python with deep knowledge of Scrapy, Selenium, and requests/BeautifulSoup. Experience with mobile app traffic analysis using proxies/sniffers. Understanding of RESTful APIs, HTTP methods, and JSON/XML formats. Familiarity with proxy services, rotating residential/datacenter IPs, and anti-bot evasion. Solid grasp of HTML, DOM parsing, and browser automation. Hands-on with Git, Linux command line, and virtual environments. Preferred Skills: Experience with tools like mitmproxy, Charles Proxy, Burp Suite. Familiarity with headless browsers like Puppeteer, Playwright (optional). Ability to reverse engineer API calls from Android/iOS apps. Knowledge of Docker, cloud deployment (AWS/GCP), and job schedulers. Basic understanding of captcha-solving services (2Captcha, CapMonster, etc.). Bonus Points: Built crawlers for domains like e-commerce, travel, social media, or financial data. Experience with CI/CD pipelines for automated crawling workflows. Knowledge of data cleaning, ETL, or streaming pipelines (Kafka, Airflow, etc.).