Pyspark Developer

Role & responsibilities

Design and Develop Data Pipelines: Create, maintain, and optimize scalable Extract,

Transform, Load (ETL) and ELT pipelines using PySpark on a distributed computing

environment (e.g., Databricks, AWS EMR, Azure Synapse).

• Code Migration and Modernization: Lead the effort to re-engineer existing Python-based data

processes, functions, and analytical logic (including legacy systems or complex Pandas

transformations) into efficient and performant PySpark code.

• Performance Tuning: Profile, optimize, and fine-tune PySpark jobs for maximum speed and

efficiency, focusing on techniques like partitioning, caching, broadcast variables, and query

optimization to handle terabyte-scale healthcare datasets.

• Data Quality and Governance: Implement robust data validation, cleansing, and monitoring

procedures within PySpark jobs to ensure the highest quality and integrity of claims, member,

and provider data.