Role: Lead Software Engineer - Data MiningLocation: Sector 65, Gurgaon, India
Job Summary
We are seeking a highly experienced and technically profound Senior Developer to lead the design, development, and maintenance of our high-value web scraping platform, specifically targeting dynamic data from the airlines, hotels, and travel aggregation industries. This role requires mastery in overcoming sophisticated anti-bot countermeasures like Kasada, Akamai (Bot Manager), and Cloudflare (Bot Management/WAF) to reliably scrape miles, points availability, pricing, and award inventory.You will be responsible for the full lifecycle: from conducting low-level reverse engineering to architecting a scalable, high-availability infrastructure on AWS. The ideal candidate blends deep expertise in adversarial web scraping with high proficiency across multiple modern tech stacks.
Key Responsibilities
Anti-Bot Bypass & Reverse Engineering Leadership:
- Adversarial Research: Proactively research, analyze, and implement persistent evasion techniques against cutting-edge bot detection systems specifically deployed by major travel/hospitality vendors (e.g., airline booking engines, hotel chains).
- Deep JavaScript Reverse Engineering: Conduct static and dynamic analysis of heavily obfuscated JavaScript sensor scripts to defeat fingerprinting vectors and identify the complex token generation logic used by Kasada and Akamai.
- Browser Internals Mastery: Implement advanced methods to hide automated signals within headless browsers (e.g., Playwright/Puppeteer), including patching browser features (navigator.webdriver, WebGL, Canvas, and timing functions).
- Low-Level Protocol Spoofing: Design and implement solutions to spoof network-level browser signatures, including TLS/JA3/JA4, HTTP/2 SETTINGS, and pseudo-header ordering to maintain consistent session authenticity.
Core Framework & High-Concurrency Development
- Advanced Behavioral Mimicry: Develop and tune sophisticated behavioral models that accurately simulate human interaction patterns across complex booking interfaces, including realistic non-linear mouse movements (Bézier curves), natural typing dynamics, and multi-page session warm-ups (crucial for defeating Captcha/behavioral analysis).
- SPA/AJAX Expertise: Build robust solutions for scraping dynamic data from modern Single Page Applications (SPAs) often used for travel searches, ensuring all necessary AJAX calls and hidden API endpoints are correctly identified and utilized.
- Cross-Stack Frameworks: Design, code, and maintain scalable, high-concurrency scraping frameworks, leveraging the strengths of Node.js/TypeScript (for browser automation/JS RE), Python (for data processing/Scrapy), or Go/Java/.NET (for
high-speed request processing).
- API Development: Build and maintain high-performance, low-latency REST API endpoints for internal teams to access the generated miles/points data.
AWS Infrastructure & Scalability
- Architecture & Deployment: Lead the design and implementation of a scalable, resilient, and cost-optimized scraping infrastructure on AWS (or comparable cloud environment).
- Containerization & Orchestration: Implement and manage containers using Docker and orchestrate high-concurrency scraping jobs using Kubernetes (EKS/AKS/GKE) or serverless technologies (AWS Fargate/Lambda) for elastic scaling.
- IP/Proxy Management: Design and maintain a sophisticated proxy orchestration system for managing rotating residential and mobile IPs, including rigorous health checks, failure detection, and automatic IP rotation/backoff strategies.
- Monitoring & Alerting: Implement robust monitoring (e.g., Prometheus/Grafana, ELK Stack) for real-time scrape health, block rates, token generation success, and infrastructure performance to ensure maximum uptime and data freshness.
Qualifications & Experience
Education & Competencies:
- Experience: 8+ years of software development experience, with a minimum of 6 years dedicated to adversarial web scraping in a production environment.
- Domain Expertise: Proven experience scraping data from major airlines, hotel groups, or meta-search travel sites—demonstrating familiarity with common industry defenses and booking flows.
- Anti-Bot Proficiency: Proven, hands-on experience successfully bypassing at least two major commercial anti-bot services (Akamai, Kasada, Cloudflare, PerimeterX, DataDome).
- Reverse Engineering Proficiency: Deep proficiency in JavaScript reverse engineering and using advanced browser debugging techniques to analyze and manipulate highly obfuscated code.
- Tech Stacks: Strong production-level experience in at least two of the following: Node.js/TypeScript, Python (Scrapy/Playwright), Go, or Java/.NET.
- Web Protocol Mastery: Expert knowledge of HTTP/2, TLS handshake, and JA3/JA4 fingerprinting. Experience using specialized TLS client libraries.
- Cloud & DevOps: Expert experience with AWS services (EKS/Fargate, Lambda) and modern deployment practices (Docker, Kubernetes, CI/CD).
Skills: api,reverse engineering,bot,aws,infrastructure
