Data Engineer

As a Data Engineer in our company, your role will involve designing and implementing data migration, ingestion, processing, and analytics pipelines using a variety of technologies including AWS (S3, Glue, Lambda, Athena, RDS Aurora, Databricks) and Azure (ADF, ADLS, Synapse, Databricks). You will be expected to apply your strong expertise in SQL, data modeling, schema design, and data quality management. Additionally, you will need to ensure governance, security, compliance, metadata management, MDM, and data catalog practices are followed. Working with big data, NoSQL, relational DBs, and data warehouse solutions will be part of your responsibilities. Building real-time pipelines with Kafka/Event Hubs and enabling Data Fabric architecture with APIs, data virtualization, and lineage tools (Collibra, Informatica, Alation, Purview) will also be a key aspect of your role. Supporting CI/CD, DevSecOps, IaC (Terraform/Bicep), and Agile Scrum practices will round out your responsibilities. Key Responsibilities: - Design and implement data migration, ingestion, processing, and analytics pipelines using AWS and Azure technologies - Apply expertise in SQL, data modeling, schema design, and data quality management - Ensure governance, security, compliance, metadata management, MDM, and data catalog practices - Work with big data, NoSQL, relational DBs, and data warehouse solutions - Build real-time pipelines with Kafka/Event Hubs and enable Data Fabric architecture with APIs, data virtualization, and lineage tools - Support CI/CD, DevSecOps, IaC (Terraform/Bicep), and Agile Scrum practices Qualifications Required: - Lead Data Engineer: 8-10 years; Data Engineer: 6-8 years of experience - Proficiency in Apache Spark (PySpark/Scala), SQL, data pipelines, and cloud migration - Strong experience with Azure and AWS data services - Familiarity with metadata management, lineage, governance tools, and schema registry - Good communication, teamwork, and problem-solving skills In addition, having knowledge of microservices, event-driven architectures, and programming languages such as Node.js, Python, or .NET would be preferred. Experience with SSIS, Hadoop ecosystem (Hive, HBase, Cassandra, Spark, Flume, Impala) is also a plus.,