Senior Data Engineer

Role Overview: As a Senior Data Engineer at Milestone Technologies, you will be responsible for leading data ingestion, transformation, and optimization initiatives using the modern Apache and Azure data stack. Your role will involve working on scalable pipelines, large-scale distributed systems, and data lake management. Core Responsibilities: - Build and manage high-volume data pipelines using Spark/Databricks. - Implement ELT frameworks using Azure Data Factory/Synapse Pipelines. - Optimize large-scale datasets in Delta/Iceberg formats. - Implement robust data quality, monitoring, and governance layers. - Collaborate with Data Scientists, Analysts, and Business stakeholders. Technical Stack: - Big Data: Apache Spark, Kafka, Hive, Airflow, Hudi/Iceberg. - Cloud: Azure (Synapse, ADF, ADLS Gen2), Databricks, AWS (Glue/S3). - Languages: Python, Scala, SQL. - Storage Formats: Delta Lake, Iceberg, Parquet, ORC. - CI/CD: Azure DevOps, Terraform (infra as code), Git. Qualifications Required: - Experience working with Apache and Azure data stack. - Proficiency in building and managing high-volume data pipelines. - Strong knowledge of ELT frameworks and data optimization techniques. - Familiarity with data quality, monitoring, and governance practices. - Excellent collaboration skills with Data Scientists, Analysts, and Business stakeholders. Our Commitment To Diversity & Inclusion: At Milestone Technologies, we are committed to creating a workplace that values diversity and inclusion. We believe in empowering all employees to bring their full, authentic selves to work. We understand that fostering a diverse and inclusive culture is essential for our success and strive to champion equity and belonging for all. Milestone Technologies provides equal employment opportunities for all applicants and employees.,