Data Collector

Data Scraping, MongoDB, Solr / ElasticSearch

We are seeking a skilled Python Developer with strong experience in web/data scraping and working knowledge of MongoDB, Solr, and/or ElasticSearch. You will be responsible for developing, maintaining, and optimizing scalable scraping scripts to collect structured and unstructured data, efficiently manage it in MongoDB, and index it for search and retrieval using Solr or ElasticSearch.

• Design and develop robust web scraping solutions using Python (e.g., Scrapy, BeautifulSoup, Selenium, etc.).
• Extract and process large volumes of data from websites, APIs, and other digital sources.
• Ensure scraping mechanisms are efficient, resilient to site changes, and compliant with best practices.
• Store, retrieve, and manage scraped data efficiently in MongoDB databases. •Index, manage, and optimize data search capabilities using Solr or ElasticSearch. •Build data validation, cleaning, and transformation pipelines.
• Handle challenges like CAPTCHA solving, IP blocking, and dynamic content rendering.
• Monitor scraping jobs and troubleshoot errors and bottlenecks.
• Optimize scraping speed, search indexing, storage efficiency, and system scalability.
• Collaborate with product managers to define data requirements.

Job Type: Full-time

Pay: ₹20,000.00 - ₹25,000.00 per month

Application Question(s):

• How immediate you can join ?

Location:

• Mumbai, Maharashtra (Required)

Work Location: In person