Job
Description
About Position: We are looking for a Senior Data Engineer to play a key role in building, optimizing, and maintaining our Azure-based data platform, which supports IoT data processing, analytics, and AI/ML applications. As part of our Data Platform Team, you will design and develop scalable data pipelines, implement data governance frameworks, and ensure high-performance data processing to drive digital transformation across our business. Responsibilities: Data Pipeline Development: Design, build, and maintain high-performance, scalable ETL/ELT pipelines using Azure Data Factory, Databricks, and ADLS. Data Platform Enhancement: Contribute to the development and optimization of our Azure-based data platform, ensuring efficiency, reliability, and security. IoT & High-Volume Data Processing: Work with large-scale IoT and operational datasets, optimizing data ingestion, transformation, and storage. Data Governance & Quality: Implement data governance best practices, ensuring data integrity, consistency, and compliance. Performance Optimization: Improve query performance and storage efficiency for analytics and reporting use cases. Collaboration: Work closely with data scientists, architects, and business teams to ensure data availability and usability. Innovation & Automation: Identify opportunities for automation and process improvements, leveraging modern tools and technologies. Requirement: 6+ years of experience in data engineering with a focus on Azure cloud technologies. Strong expertise in Azure Data Factory, Databricks, ADLS, and Power BI. Proficiency in SQL, Python, and Spark for data processing and transformation. Experience with IoT data ingestion and processing, handling high-volume, real-time data streams. Strong understanding of data modeling, lakehouse architectures, and medallion frameworks. Experience in building and optimizing scalable ETL/ELT processes. Knowledge of data governance, security, and compliance frameworks. Experience with monitoring, logging, and performance tuning of data workflows. Strong problem-solving and analytical skills with a platform-first mindset.
