Data Engineer Scala/Pyspark

Roles and Responsibilities

• Develops and maintains scalable data pipelines in response to customer requirements.
• Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
• Design, develop, implement and run data solutions that improve data efficiency, reliability and quality.
• Implements unit test cases using PyTest and perform quality checks.
• Ensure the code is meeting the desired quality checks using Sonar.
• To make sure all the cloud infra is intact and resolve any issues encountered.
• Deploy and execute code using AKS and making sure all configs are in place.
• Schedule the deployed pipelines using Airflow following proper dependency hierarchy of jobs.
• Move the code/application to higher environments(stage/prod) from non-prod for go-live.
• Maintain all credentials using Azure Key Vault service to ensure security.
• Build and maintain pipelines and automation through GitOps (GitHub Actions, Jenkins, etc).
• Identifies solutions to non-standard complex requests and problems and creates solution using available technologies.

Education & Experience

• Bachelor/masters degree in computer science or information technology or equivalent with 4 to 6 years of work experience.
• Solid experience in programming languages like Python, Scala.
• Good experience in job schedulers like Airflow.
• Expertise in BigData frameworks like Spark and good understanding of Hadoop concepts.
• Good experience in working on cloud environment, preferably Azure.
• Proven experience in cloud services like Azure Kubernetes, Azure Data Lake, Azure Databricks, Azure Container Registries, Azure Service Bus, Azure Key Vault.
• Solid experience and expertise in writing complex queries in SQL and PL/SQL.
• Experience in writing Shell scripting and Snowflake.
• Experience with Continuous Integration CI/CD pipelines using Jenkins, GitHub Actions.
• Experience working in an agile environment.
• Experience working across teams and proven track record in solution focused problem-solving skills.
• jdGood understanding of US healthcare will be an added advantage.