Threat Intelligence Data Engineer – Automated Collection & Dark Web Intelligence

About the Role

achieve near-total coverage of global breach and leak data with 99%+ automation

What You Will Do:

Automated Intelligence Collection & Discovery

• large-scale, distributed crawling and discovery systems

• Surface web, deep web, and dark web
  

• Hacker forums, underground marketplaces, and breach communities
  

• Chat platforms (Telegram, Discord, IRC, WhatsApp, etc.)
  

• Paste sites, code repositories, and social platforms used for breach disclosure
  

• Continuously discover, archive, and download newly released datasets, logs, credentials, and artifacts the moment they appear
  

Dark Web, Anonymized & Decentralized Networks

• Build automated collectors and archivers for anonymized and decentralized networks including:
  

• Tor (.onion), I2P, ZeroNet, Freenet, IPFS, GNUnet, Lokinet, Yggdrasil, and similar systems
  

• Design resilient workflows for unreliable, adversarial, or ephemeral data sources
  

• Normalize and index data from non-traditional network protocols and formats
  

Infrastructure & Exposure Discovery

• Develop automated scanning systems to identify:
  

• Unsecured databases (Elasticsearch, MySQL, PostgreSQL, MongoDB, etc.)
  

• Exposed cloud storage (S3, Azure, GCP, DigitalOcean Spaces)
  

• Open FTP servers, backups, and misconfigured archives
  

• Monitor and ingest data from file hosting and distribution platforms commonly used for breach dumps
  

Pipeline Engineering & Operations

• Build ETL pipelines to clean, normalize, enrich, and index structured and unstructured data
  

• anti-bot evasion strategies

• Integrate collected intelligence into centralized databases and search systems
  

• Design APIs and internal tooling to support downstream analysis and AI/ML workflows
  

• Implement advanced anti-bot, evasion, and resiliency techniques (proxy rotation, fingerprinting, CAPTCHA mitigation, session handling)
  

• Automate deployment, scaling, and monitoring using Docker, Kubernetes, and cloud infrastructure
  

• Continuously optimize performance, reliability, and cost efficiency of crawler clusters
  

• data engineering, intelligence collection, crawling, or distributed data pipelines

• Strong Python expertise and experience with frameworks such as Scrapy, Playwright, Selenium, or custom async systems
  

• high-volume, automated data collection systems in production

• Deep understanding of web protocols, HTTP, DOM parsing, and adversarial scraping environments
  

• Experience with asynchronous, concurrent, and distributed architectures
  

• Familiarity with SQL and NoSQL databases (PostgreSQL, MongoDB, Elasticsearch, Cassandra)
  

• Strong Linux/Unix, shell scripting, and Git-based workflows
  

• Experience deploying and operating systems using Docker, Kubernetes, AWS, or GCP
  

• Excellent analytical, debugging, and problem-solving skills
  

• Strong written and verbal communication skills.
  

Preferred / High-Value Experience

• dark web intelligence, breach data, OSINT, or threat research

• Familiarity with Tor, I2P, underground forums, stealer logs, or credential ecosystems
  

• Experience processing large breach datasets or stealer logs
  

• adversarial data environments

• Exposure to AI/ML-driven intelligence platforms
  

• Position Type:

  Full-time

• Location:

  Remote in India.

  Work from wherever you please! Your home, the beach, our offices, etc.

• Compensation:

  USD 1300-2000 monthly

• Professional Growth:

  Amazing upward mobility in a rapidly expanding company.

• Innovative Culture:

  Be part of a team that leverages AI and cutting-edge technologies.

About Us: