Azure & Big Data - Lead

Key Responsibilities:

Design, develop, and deploy scalable batch and streaming data pipelines using PySpark, Flink, Scala, SQL, and Redis. Lead migration of complex on-premise workflows to Azure cloud ecosystem (Databricks, ADLS, Azure Data Factory), optimizing infrastructure and deployment processes. Implement performance tuning strategies to reduce job runtimes and enhance data reliability, including optimization of Unity Catalog tables. Collaborate with product stakeholders to deliver high-priority data features and ensure alignment with business goals. Manage and mentor an 8-member offshore team, fostering best practices in data engineering and agile development. Conduct internal training sessions on modern data architecture, cloud-native deployments, and data engineering best practices.

Required Skills & Technologies: Big Data Tools: PySpark, Spark, Flink, Hive, Hadoop, Delta Lake, Streaming, ETL Cloud Platforms: Azure (ADF, Databricks, ADLS, Event Hub), AWS (S3) Orchestration & DevOps: Airflow, Docker, Kubernetes, GitHub Actions, Jenkins Programming Languages: Python, Scala, SQL, Shell Other Tools: Redis, Solace, MQ, Kafka, Grafana, Postman Soft Skills: Team Leadership, Agile Methodologies, Stakeholder Management, Technical Training

Certifications (Good to have): Databricks Certified: Data Engineer Associate, Lakehouse Fundamentals Microsoft Certified: Azure Fundamentals (AZ-900), Azure Data Fundamentals (DP-900)

Preferred Qualifications: Bachelors degree in Engineering (E.C.E.) with strong academic performance.Proven experience in financial data pipelines, regulatory reporting, and risk analytics