Azure Data Engineer - ETL/Data Pipeline

Key Responsibilities

• Design and develop scalable real-time data streaming solutions using Apache Kafka and Python.
• Architect and implement ETL/ELT pipelines using Azure Databricks for both structured and unstructured healthcare data.
• Optimize and maintain Kafka applications, Python scripts, and Databricks workflows to ensure performance and reliability.
• Ensure data integrity, security, and compliance with healthcare standards such as HIPAA and HITRUST.
• Collaborate with data scientists, analysts, and business stakeholders to gather requirements and translate them into robust data solutions.
• Mentor junior engineers, perform code reviews, and promote engineering best practices.
• Stay current with evolving technologies in cloud, big data, and healthcare data standards.
• Contribute to the development of CI/CD pipelines and containerized environments (Docker, Kubernetes).
  

Required Skills & Qualifications

• 4+ years of hands-on experience in data engineering roles.
• Strong proficiency in Kafka (including Kafka Streams, Kafka Connect, Schema Registry).
• Proficient in Python for data processing and automation.
• Experience with Azure Databricks (or readiness to ramp up quickly).
• Solid understanding of cloud platforms, with a preference for Azure (AWS/GCP is a plus).
• Strong knowledge of SQL and NoSQL databases; data modeling for large-scale systems.
• Familiarity with containerization tools like Docker and orchestration using Kubernetes.
• Exposure to CI/CD pipelines for data applications.
• Prior experience with healthcare datasets (EHR, HL7, FHIR, claims data) is highly desirable.
• Excellent problem-solving abilities and a proactive mindset.
• Strong communication and interpersonal skills to work in cross-functional teams.