Job
Description
As a Senior Data Engineer in the healthcare domain, your role involves architecting, developing, and managing large-scale, secure, and high-performance data pipelines on Databricks using Spark, Delta Lake, and cloud-native tools. You will be responsible for designing and implementing healthcare-specific data models to support analytics, AI/ML, and operational reporting. Your expertise will be crucial in ingesting and transforming complex data types such as 837/835 claims, EHR/EMR records, provider/member files, lab results, and clinical notes. Additionally, you will lead data governance, quality, and security initiatives ensuring compliance with HIPAA, HITECH, and organizational policies. Collaboration with cross-functional stakeholders to understand data needs and provide robust, scalable solutions will be a key aspect of your role. Mentoring junior and mid-level engineers, identifying performance bottlenecks, and implementing optimizations in Spark jobs and SQL transformations are also part of your responsibilities. You will own and evolve best practices for CI/CD, version control, and deployment automation, while staying up to date with industry standards and evaluating new tools/technologies. Key Responsibilities: - Architect, develop, and manage large-scale, secure, and high-performance data pipelines on Databricks using Spark, Delta Lake, and cloud-native tools. - Design and implement healthcare-specific data models to support analytics, AI/ML, and operational reporting. - Ingest and transform complex data types such as 837/835 claims, EHR/EMR records, provider/member files, lab results, and clinical notes. - Lead data governance, quality, and security initiatives ensuring compliance with HIPAA, HITECH, and organizational policies. - Collaborate with cross-functional stakeholders to understand data needs and provide robust, scalable solutions. - Mentor junior and mid-level engineers, performing code reviews and technical guidance. - Identify performance bottlenecks and implement optimizations in Spark jobs and SQL transformations. - Own and evolve best practices for CI/CD, version control, and deployment automation. - Stay up to date with industry standards (e.g., FHIR, HL7, OMOP) and evaluate new tools/technologies. Required Qualifications: - 5+ years of experience in data engineering, with 3+ years in healthcare or life sciences domain. - Deep expertise with Databricks, Apache Spark (preferably PySpark), and Delta Lake. - Proficiency in SQL, Python, and data modeling (dimensional/star schema, normalized models). - Strong command over 837/835 EDI formats, CPT/ICD-10/DRG/HCC coding, and data regulatory frameworks. - Experience with cloud platforms such as Azure, AWS, or GCP and cloud-native data services (e.g., S3, ADLS, Glue, Data Factory). - Familiarity with orchestration tools like Airflow, dbt, or Azure Data Factory. - Proven ability to work in agile environments, manage stakeholder expectations, and deliver end-to-end data products. - Experience implementing monitoring, observability, and alerting for data pipelines. - Strong written and verbal communication skills for both technical and non-technical audiences.,
