Python Developer

Python Developer Data Scraping, MongoDB, Solr / ElasticSearch

We are seeking a skilled Python Developer with strong experience in web/data scraping and working knowledge of MongoDB, Solr, and/or ElasticSearch. You will be responsible for developing, maintaining, and optimizing scalable scraping scripts to collect structured and unstructured data, efficiently manage it in MongoDB, and index it for search and retrieval using Solr or ElasticSearch.

Key Responsibilities:

• Design and develop robust web scraping solutions using Python (e.g., Scrapy, BeautifulSoup, Selenium, etc.).

• Extract and process large volumes of data from websites, APIs, and other digital sources.

• Ensure scraping mechanisms are efficient, resilient to site changes, and compliant with best practices.

• Store, retrieve, and manage scraped data efficiently in MongoDB databases.

• Index, manage, and optimize data search capabilities using Solr or ElasticSearch.

• Build data validation, cleaning, and transformation pipelines.

• Handle challenges like CAPTCHA solving, IP blocking, and dynamic content rendering.

• Monitor scraping jobs and troubleshoot errors and bottlenecks.

• Optimize scraping speed, search indexing, storage efficiency, and system scalability.

• Collaborate with product managers to define data requirements.

Required Skills and Qualifications:

• 3+ years of experience with Python, specifically in web scraping projects.

• Proficient in scraping libraries such as Scrapy, BeautifulSoup, Requests, Selenium, or similar.

• Hands-on experience with MongoDB (querying, indexing, schema design for unstructured/structured data).

• Strong experience with Solr or ElasticSearch for data indexing, search optimization, and querying.

• Good understanding of HTML, CSS, XPath, and JSON.

• Experience handling anti-scraping mechanisms like IP rotation, proxy usage, and headless browsers.

• Familiarity with RESTful APIs and parsing data formats like JSON, XML, CSV.

• Strong problem-solving skills and attention to detail.

• Good written and verbal communication skills.