Associate - Data Engineer

As a Data Engineer, you will be responsible for managing and optimizing data pipelines for efficient data processing. Your role will involve working with a variety of technologies and platforms to ensure smooth data workflows. Key Responsibilities: - Utilize your 5+ years of experience in data engineering, including at least 2+ years of hands-on experience with Databricks. - Demonstrate strong expertise in Apache Spark using PySpark, Scala, and SQL for efficient distributed data processing. - Implement and maintain Delta Lake and Lakehouse architecture for effective data modeling. - Work with cloud platforms such as Azure Data Lake, AWS S3, or GCP BigQuery/Storage to manage data infrastructure. - Utilize your proficiency in SQL for data manipulation and performance tuning to optimize data processing tasks. - Collaborate with ETL frameworks and workflow orchestration tools like Airflow, ADF, and DBX Workflows for seamless data workflows. - Implement CI/CD practices, Git-based workflows, and DevOps principles to ensure efficient data pipeline management. - Exposure to MLOps and MLflow will be a strong plus to enhance machine learning operationalization processes. - Apply your knowledge of data governance, cataloging, and security frameworks to maintain data integrity and compliance. Qualifications Required: - 5+ years of experience in data engineering, with at least 2+ years of hands-on experience with Databricks. - Strong expertise in Apache Spark (PySpark/Scala/SQL) and distributed data processing. - Solid experience with Delta Lake, Lakehouse architecture, and data modeling. - Hands-on experience with at least one cloud platform: Azure Data Lake, AWS S3, or GCP BigQuery/Storage. - Proficiency in SQL for data manipulation and performance tuning. - Experience with ETL frameworks, workflow orchestration tools (Airflow, ADF, DBX Workflows). - Understanding of CI/CD, Git-based workflows, and DevOps practices. - Exposure to MLOps and MLflow is a strong plus. - Knowledge of data governance, cataloging, and security frameworks. (Note: Additional details about the company were not present in the provided job description.),