Module Lead / Technical Lead - Azure Data Engineer

Job Description

Primary Skills (Must have experience): Python package builds(setup,poetry, wheels,artifact registries) Specific technologies Databricks(PySpark,Databricks Asset Bundles) Open File Formats (Delta/Parquet/Iceberg/etc) SQL Databases (Unity Catalog, SQL Server, Hive, Postgres) Orchestration Tools(Databricks Workflows, Airflow, Dagster) Azure Data Lake Storage Azure Active Directory (AD groups, Service Principles,Managed Identities) Role & responsibilities We are seeking a highly skilled Data Engineering Specialist to join our team. The ideal candidate will have extensive experience in cloud technologies, DevOps development practices and Data Engineering to support and enhance initiatives. ________________________________________ Key Responsibilities: Design, develop, and maintain Databricks Lakehouse solutions sourcing data from cloud platforms such as GCP, ERPs sources, Relational stores, NoSQL and on-prem sources. Develop and optimize data pipelines and workflows in Databricks, utilizing PySpark, SparkSQL and Databricks Asset Bundles. Govern and manage data with Unity Catalog , Azure SQL, Hive, Postgres. Implement orchestration solutions using Azure data factory, Databricks Workflows, Airflow, and Dagster. Work with event-driven architectures using Kafka, Azure Event Hub, Google Cloud Pub/Sub. Develop and maintain Change Data Capture (CDC) solutions using tools like Debezium. Extensive experience in design and implementation of data migration projects specifically involving Azure Synapse and Databricks Lakehouse Manage cloud storage solutions, including Azure Data Lake Storage and Google Cloud Storage. Build, manage, and optimize Python packages using poetry, wheels, and artifact registries. Implement and manage DevOps , CI/CD pipelines with GitHub, Git Actions for efficient version control and release process Apply best practices in test-driven development, code review, branching strategies, and deployment processes. Configure and manage identity & access solutions using Azure Active Directory, including AD Groups, Service Principals, and Managed Identities. If you are interested , please share your updated resume with us.