Lead Data Crawling Engineer

Position: Lead Data Crawling Engineer

Location: Sector 23 Dwarka, Delhi

This is a Delhi-based position and work from office only!!

Job Summary:

Key Responsibilities:

• Design, develop, and maintain

  complex, large-scale web crawling and scraping systems

  to extract structured and unstructured data.
• Build

  high-performance automated pipelines

  using Selenium, Scrapy, Beautiful Soup, Requests, and Python-based frameworks.
• Handle

  dynamic, JavaScript-heavy websites

  using tools like Selenium, Playwright, or headless browsers.
• Develop and optimize

  distributed crawling setups

  , load balancing, and scalable architectures.
• Implement

  API-based data extraction

  including REST, GraphQL, OAuth, and rate-limit management.
• Clean, preprocess, validate, and store data efficiently using

  Pandas, SQL databases

  , and cloud storage solutions.
• Monitor, debug, and improve pipeline performance, ensuring high uptime and consistent data quality.
• Collaborate with analytics, engineering, and product teams to deliver data required for reporting and machine learning needs.
• Ensure all scraping activities follow

  ethical standards, legal guidelines, and robots.txt compliance

  .
• Mentor junior developers and contribute to coding best practices and architecture decisions.

Required Skills & Qualifications:

• Experience:

  4–6 years of professional experience in data crawling, scraping automation, and large-scale data extraction.

• Strong proficiency in Python

  , with hands-on experience using:

a. Selenium

b. Scrapy

c. Beautiful Soup

d. Requests

• Excellent understanding of

  HTML, CSS, JavaScript, HTTP protocols

  , and browser behavior.
• Strong experience with

  data manipulation

  using Pandas.
• Expertise in

  SQL

  , database design, indexing, and performance optimization.
• Strong understanding of

  API integrations

  , authentication methods, and data exchange formats.
• Experience with

  Git

  , CI/CD workflows, and collaborative development environments.

Nice-to-Have Skills:

• Experience with

  cloud-based scraping

  tools or serverless functions (AWS Lambda, GCP Cloud Functions, Azure Functions).
• Familiarity with

  distributed crawling

  , proxy rotation, headless browser automation, and captcha solving techniques.
• Experience handling

  large-scale data collection

  , data warehousing, or pipeline orchestration tools (Airflow, Prefect, Luigi).
• Knowledge of

  containerization (Docker)

  and cloud deployment.
• Experience with

  log management and monitoring tools

  (ELK, Grafana, Prometheus).

Soft Skills:

• Strong problem-solving abilities with a keen eye for detail.
• Ability to write

  clean, scalable, and maintainable code

  .
• Excellent debugging and performance optimization skills.
• Strong communication and stakeholder management abilities.
• Ability to lead small projects and mentor junior team members.

Nice to Have (Python & Advanced Technical Skills):

• Experience with

  cloud-based scraping tools or serverless services

  such as AWS Lambda, Google Cloud Functions, or Azure Functions.
• Familiarity with

  distributed crawling architectures

  , parallel scraping, proxy management, and data pipeline orchestration (e.g., Airflow, Prefect, Luigi).
• Hands-on experience in

  large-scale data collection, processing, and storage

  systems (data lakes, warehousing, cloud storage, etc.).
• Understanding of

  ethical, compliant, and legally safe web scraping practices

  , including robots.txt, rate limits, and data protection guidelines.

About Nuvoretail (

Nuvoretail Enlytical Technologies Private Limited is an e-commerce analytics and automation company. Our proprietary digital shelf analytics and automation platform called Enlytical.ai helps e-commerce brands solve the complexities in today’s e-commerce landscape by offering a unified and all- encompassing business view on the various aspects of e-commerce business. Our platform leverages insights drawn from multiple data points that help our clients win in e-commerce by gaining a competitive edge with data-driven insights for sharper decision-making. The insights cover all aspects of e-commerce such as digital product portfolio analysis, supply chain analytics, e-commerce operations automation, pricing, and competitor benchmarking, and Amazon advertising automation using our proprietary algorithms.

As a leading e-commerce service provider, we offer the most comprehensive end-to-end e-commerce solutions to brands, both in India and abroad. Right from preparing a road map to writing our client’s e- commerce success story to assisting them In increasing their online sales, we do everything via our diverse e-commerce services and bespoke strategies and technology. Our services span across the brand’s e-commerce enablement including content and digital asset creation for product listing, On Platform, and Off Platform marketing services with deep expertise in Amazon Marketing Services (AMS), Amazon SEO through keyword research, e-Commerce operations across various e-commerce platforms, website development, social media marketing, and AI-enabled e-Commerce MIS Dashboards.

Awards & Recognition:

CIOReviewIndia