Senior Data Engineer

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.
Our seasoned professionals deliver services based on Milestone’s best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Senior Data Engineer - Responsibilities (7+ Years Exp)

• Build scalable data ingestion pipelines for relational systems and unstructured/semi-structured sources.
• Design and optimize lakehouse workloads using Apache Iceberg.
• Optimize table layouts, partitioning strategies, compaction, and schema evolution for Iceberg-based datasets.
• Implement best practices for versioning, time travel, incremental processing, and ACID guarantees.
• Develop and tune Apache Spark (batch & streaming) for performant, scalable data processing.
• Work with AWS services (Glue, EMR, Lambda, Step Functions, S3) focusing on cost optimization and performance tuning.
• Implement and manage Kafka and Kafka streaming pipelines for large-scale real-time processing.
• Build and orchestrate data workflows using DBT and Airflow.
• Leverage AI/LLM automation to accelerate ETL/ELT development, validation, and debugging.
• Implement observability—monitoring, logging, and alerting—to ensure platform stability and reliability.
• Participate in code reviews to ensure code quality and share best practices with the team.

Required Skills

• 7 Years of relevant experience
• Strong SQL, Python, and Spark experience
• Deep experience ingesting and processing structured & unstructured data
• Strong knowledge of Apache Iceberg and lakehouse architectures
• Hands-on AWS experience with tuning and cost optimization
• Expertise with Kafka, Kafka Streaming, DBT, and Airflow
• Solid understanding of distributed systems and data engineering best practices

Estimated Pay Range:
Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.