Senior Data Engineer

Key Responsibilities

• Architect and implement end-to-end data pipelines using Azure Data Factory, Databricks, and Snowflake for large-scale data ingestion, transformation, and storage.
• Using Microsoft Azure data PaaS services, design, build, modify, and support data pipelines leveraging DataBricks and PowerBI in a medallion architecture setting.
• If necessary, create prototypes to validate proposed ideas and solicit input from stakeholders.
• Excellent grasp of and expertise with test-driven development and continuous integration processes.
• Analysis and Design Converts high-level design to low-level design and implements it.
• Collaborate with Team Leads to define/clarify business requirements, estimate development costs, and finalize work plans.
• Run unit and integration tests on all created code Create and run unit and integration tests throughout the development lifecycle.
• Benchmark application code proactively to prevent performance and scalability concerns.
• Collaborate with the Quality Assurance Team on issue reporting, resolution, and change management.
• Support and Troubleshooting Assist the Operations Team with any environmental issues that arise during application deployment in the Development, QA, Staging, and Production environments.
• Familiarity with PowerBI and Reltio is advantageous but not required.
• Collaborate with BI teams to ensure data models are optimized for reporting in Power BI, with a focus on performance and usability.
• Establish data governance, quality, and security controls, ensuring compliance with GDPR, HIPAA, and global clinical data regulations.
• Mentor and guide junior engineers, fostering technical excellence and knowledge sharing.
• Drive automation and CI/CD practices within data engineering pipelines, integrating with version control and deployment workflows.
• Work closely with Data Architects, Business Analysts, and Product Owners to translate business needs into technical solutions.
  

Skills

• Expert-level knowledge of Azure Data Factory, Databricks, and Snowflake.
• Understanding of quality processes and estimate methods.
• Understanding of design concepts and architectural basics.
• Fundamental grasp of the project domain.
• The ability to transform functional and nonfunctional needs into system requirements.
• The ability to develop and code complicated applications is required.
• The ability to create test cases and scenarios based on specifications.
• Solid knowledge of SDLC and agile techniques.
• Knowledge of current technology and trends.
• Logical thinking and problem-solving abilities, as well as the capacity to collaborate.
• Primary skills: Cloud Platform, Azure, Databricks, ADF, ADO.
• Sought: SQL, Python, PowerBI.
• General Knowledge: PowerApps, Java/Spark, Reltio.
• 5-7 years of experience in software development with minimum 3 years of cloud computing.
• Proficient in SQL, Python, and cloud-native architecture.
• Strong grasp of data security, privacy compliance, and best practices in a regulated environment.
  

Education

• Bachelor's Degree in technical discipline (Math's, Science, Engineering, Computing, etc.)