Job
Description
Job Description: Lead Data Engineer Experience Level: 7-10 years WFO-Chennai Required Skills: AI /ML-skill mandatory · AWS Glue, EMR (Spark), Redshift, Step Functions: Deep experience in using AWS tools for data processing and pipeline orchestration. · Python, SQL, PySpark/Scala: Advanced skills in Python and SQL, as well as experience in Spark-based data processing frameworks. · Data Pipeline Performance: Expertise in tuning and optimizing data pipelines for performance and scalability. · Monitoring & Troubleshooting: Experience with monitoring data systems and troubleshooting performance bottlenecks. Job Description (JD): The Lead Data Engineer will oversee the design, development, and optimization of data pipelines that serve as the backbone for data processing and analytics. The candidate should have a strong command of cloud-based data engineering tools and the ability to lead teams in building scalable, high-performance data systems. You will collaborate closely with data scientists, analysts, and product teams to deliver reliable and efficient data architectures on AWS, ensuring they meet both current and future business needs. Roles and Responsibilities: · Lead Data Pipeline Development: Architect and develop scalable, secure, and optimized data pipelines using AWS Glue, Redshift, EMR (Spark), and Step Functions. · Data Performance Tuning: Optimize data pipelines for performance, ensuring minimal latency and high throughput. · Collaborate with Stakeholders: Work with data scientists, business analysts, and other engineers to understand requirements and deliver effective solutions. · Data Storage & Management: Ensure efficient management of data storage in Redshift, Glue, and other AWS services. · Team Leadership & Mentorship: Guide and mentor a team of data engineers, ensuring best practices for data engineering are followed. · System Monitoring & Troubleshooting: Set up monitoring for all data pipelines and perform proactive troubleshooting to minimize downtime. · Continuous Improvement: Stay up-to-date with emerging technologies and improve existing data pipelines to enhance performance and scalability. Qualifications: · Bachelor’s degree in Computer Science, Engineering, or a related field. · 7+ years of experience in data engineering with expertise in AWS technologies. · Advanced skills in Python, SQL, and Spark. · Solid understanding of data engineering principles, including ETL processes, performance optimization, and scalability. · Experience in leading teams and mentoring junior engineers. · Excellent communication skills and ability to collaborate with cross-functional teams. Preferred Skills: · Experience with containerization (Docker, Kubernetes). · Familiarity with Apache Kafka, Kinesis, or other streaming technologies. · Knowledge of machine learning frameworks and their integration into data pipelines. · Familiarity with infrastructure as code tools such as Terraform. Skills PRIMARY COMPETENCY : Data Engineering PRIMARY SKILL : DBA / Data Modeling / Data Engineering PRIMARY SKILL PERCENTAGE : 100 Show more Show less
