Data Engineer

Job Description

Roles and Responsibilities Design, develop, test, deploy, and maintain large-scale data engineering solutions using Python/Pyspark on AWS EMR/Spark. Collaborate with cross-functional teams to gather requirements and design scalable data pipelines for big data processing. Develop automated testing frameworks using CICD tools to ensure high-quality code delivery. Implement CI/CD pipelines using Apache Airflow or similar technologies to manage workflows and schedules. Troubleshoot complex issues related to data processing, pipeline failures, and system performance. Desired Candidate Profile 8-13 years of experience in Data Engineering with expertise in Python/Pyspark programming language. Strong understanding of cloud-based infrastructure management (AWS) and familiarity with Azure Databricks/Spark. Experience working on large-scale projects involving big data processing using Hadoop ecosystem components such as HDFS, YARN, Oozie etc. . Proficiency in developing efficient SQL queries for querying massive datasets stored in relational databases like MySQL/Postgres/SQL Server.