Web Scraping Developer

Position:

Location:

Experience:

Vacancies:

Employment Type:

Job Summary

We are seeking a detail-oriented and experienced Web Scraping Developer with strong expertise in Python, Selenium, and Perl. The ideal candidate will be responsible for developing, maintaining, and optimizing automated data extraction tools to gather structured and unstructured data from various web sources reliably and efficiently.

Key Responsibilities

Design and implement robust and scalable web scraping solutions using Python, Selenium, and Perl.

Automate data extraction from dynamic and static websites, including those using JavaScript and AJAX.

Clean, parse, and store scraped data in structured formats (e.g., CSV, JSON, SQL, NoSQL).

Build reusable scraping scripts with error handling, logging, and scheduling.

Maintain and update scraping pipelines to adapt to website structure changes.

Monitor scraping jobs for performance, data quality, and compliance with legal/data usage policies.

Integrate scraping tools with APIs, databases, or Google Sheets as needed.

Document all code, processes, and workflows clearly and effectively.

Required Skills

Programming Languages: Strong proficiency in Python and Perl.

Frameworks/Tools: Selenium WebDriver, Beautiful Soup, Requests, Scrapy (optional), Mechanize (for Perl).

Data Handling: Experience with JSON, XML, CSV, and database interaction (MySQL, PostgreSQL, MongoDB).

Other Skills: Regex, Xpath, multithreading/asynchronous scraping.

Familiarity with browser dev tools, network sniffing, and anti-bot evasion techniques (headers, proxies, captchas).

Experience working with task schedulers (e.g., cron, Celery) and log management.

Preferred Qualifications

Bachelor's degree in Computer Science, Engineering, or related field.

Experience working with cloud platforms (AWS, Azure, or GCP) for deploying scraping jobs.

Knowledge of version control systems like Git.