Lead II - Data Engineering

7 - 9 Years
1 Opening
Trivandrum

Job Summary:

We are looking for a highly skilled Senior Data Engineer with over 6 years of experience in data engineering and a strong focus on data migration projects. The ideal candidate will have deep expertise in Azure Data Factory (ADF), ETL/ELT processes, and Python/PySpark, with proven experience building scalable and efficient data pipelines across diverse data environments.

Key Responsibilities:

• Design, develop, and implement end-to-end data pipelines across various sources and destinations.

• Lead and support data migration initiatives from legacy to modern data platforms.

• Develop robust ETL/ELT workflows using Azure Data Factory.

• Write clean, efficient, and scalable code in Python or PySpark for data transformation and automation.

• Utilize SQL for complex querying, data manipulation, and performance tuning.

• Collaborate with DevOps teams to manage CI/CD pipelines using Azure DevOps.

• Deploy and manage infrastructure using Terraform and other IaC tools.

• Ensure data solutions align with data governance, data warehousing, and dimensional modeling best practices.

• Work closely with cross-functional teams to deliver high-quality, timely data solutions.

Mandatory Skills:

• 6+ years of experience in data engineering with a focus on data migration projects.

• Hands-on expertise with Azure Data Factory (ADF).

• Strong experience in ETL/ELT process design and implementation.

• Proficiency in Python/PySpark for data manipulation and automation.

• Expert-level SQL skills.

• Solid experience with Azure DevOps for CI/CD pipeline management.

• Experience with Terraform for deploying and managing Azure infrastructure.

• Strong knowledge of data warehousing, dimensional modeling, and data governance principles.

Good-to-Have Skills:

• Experience with other Azure services (e.g., Azure Synapse, Azure Data Lake).

• Knowledge of data quality frameworks and monitoring tools.

• Familiarity with Agile/Scrum development methodologies.

• Certifications in Azure Data Engineering or related technologies.

• Exposure to big data tools and frameworks (e.g., Databricks, Spark).

Soft Skills:

• Strong analytical and problem-solving skills with attention to detail.

• Excellent communication and interpersonal skills.

• Ability to work both independently and within a team in a fast-paced environment.

Pyspark,Azure Data Factory,Etl,Terraform