Data Engineering Lead

Job Description

Data Migration & Integration: Lead data migration projects to move on-premise data (Oracle, SQL Server) to AWS cloud-based solutions (Redshift, S3, RDS). Coordinate end-to-end processes for seamless data transfers and integrations across platforms. Data Ingestion & Pipelines: Design, implement, and maintain robust, scalable data ingestion pipelines to integrate structured and unstructured data from various sources into cloud-based platforms like AWS. Use tools like Informatica, Apache Airflow, or custom Python solutions. Cloud Infrastructure (AWS): Develop cloud-native data architectures using AWS services (e.g., Redshift, S3, Glue, Lambda, RDS, and Athena). Optimize data storage, processing, and retrieval in AWS to meet performance and cost efficiency goals. ETL/ELT Development: Build and optimize ETL/ELT processes using tools like Informatica, AWS Glue, AWS Transfer Family and custom Python-based solutions. Ensure data flows are automated, efficient, and scalable. Automation & Workflow Orchestration: Implement and manage workflow orchestration with Apache Airflow to automate and schedule ETL tasks, ensuring reliable data delivery to target systems. Collaborate with Stakeholders: Work closely with business users, analysts, and other engineering teams to understand data requirements, propose data solutions, and ensure alignment with business goals. Translate business needs into technical solutions. Data Quality & Governance: Ensure data quality, integrity, and compliance with governance policies. Implement monitoring and logging to ensure pipeline health and detect anomalies or issues proactively. Mentoring & Leadership: Mentor and guide junior data engineers, helping them grow in their technical skills and best practices. Promote a culture of continuous learning and high performance. Performance Tuning & Optimization: Continuously monitor and optimize data pipeline performance, troubleshoot issues, and apply best practices for improving query performance and data processing times