Software Development Engineer 2

Key Responsibilities:

• Lead the design, development, and implementation of complex data pipelines and infrastructure. This includes utilizing cutting-edge technologies like

  Apache Beam, Spark Streaming, and Kafka for real-time data processing at scale.

• Build & maintain high tps, reliable, performant and cost-effective data collection and extraction modules using

  Node.js & Python

  , using streaming solutions like Kafka.
• Deploy, maintain and support these modules on

  AWS & GCP

  cloud.
• Index, archive and retain necessary data in multiple persistence stores like Object stores(S3), Key value store (Dynamo DB), and Elastic Search based on the use case.
• Manage the quality of data collected using data quality libraries built using

  SQL/Python/Spar

  k on AWS Glue and exposed as Dashboards for monitoring using AWS Quick sight and Kibana.
• Restfully abstract the data collected to the downstream applications through a Node.js backend.
• Collaborate well with engineers, researchers, and data implementation specialists to design.

Qualifications & Experience:

• 5+ years of experience as a Software Development Engineer

  with a proven track record of building and maintaining large-scale data processing pipelines.
• Excellent knowledge of programming languages such as Node.JS, Python
• Strong understanding of software design patterns, algorithms, and data structures.
• In-depth knowledge of

  distributed systems, cloud platforms

  (AWS, GCP), and

  streaming architectures.

• Strong understanding of

  data modeling techniques and proficiency with SQL and NoSQL databases.

• Experience with

  containerization technologies like Docker and Kubernetes (a plus).

• Experience in writing high volume/tps,

  reliable crawlers and scrapers is a plus

  .
• Excellent

  communication, collaboration, and problem-solving skills.

• Ability

  to lead and mentor junior engineer.