Senior Data Engineering Lead

Role Overview: As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing data pipelines, data lakes, and large-scale distributed systems. Your expertise in cloud platforms, ETL tools, data warehousing, and performance tuning will be crucial for this role. You should have hands-on experience with Python, PySpark, SQL, and AWS, along with exposure to big data platforms like Databricks, Snowflake, BigQuery, and Delta Lake. Key Responsibilities: - Design & Develop Pipelines: Build scalable data pipelines for ingestion, transformation, and storage from multiple sources (databases, APIs, streaming). - Data Warehousing: Work with Snowflake, BigQuery, Delta Lake, and other lakehouse architectures to ensure performance, scalability, and cost optimization. - Cloud & ETL Tools: Leverage ETL/ELT frameworks such as Apache Airflow, Informatica, Talend, AWS Glue, Dataproc, or Azure ADF. - Programming & Scripting: Write clean, efficient code in Python, PySpark, and SQL for data manipulation and transformation. - Performance Tuning: Optimize pipelines for speed, cost efficiency, and minimal resource consumption. - DevOps & Infrastructure: Collaborate with DevOps teams for CI/CD integration, IaC (Terraform/CloudFormation), monitoring, and infrastructure provisioning. - Compliance & Security: Ensure pipelines meet data governance, audit, and security standards. - Collaboration: Work with architects, analysts, and business stakeholders to translate requirements into scalable solutions. - Mentorship: Guide and mentor junior engineers, ensuring adherence to best practices, coding standards, and certifications. Qualification Required: - Strong expertise in Data Lakes, AWS, Python, PySpark, SQL. - Solid experience in distributed data processing frameworks. - Proficiency in ETL tools (Airflow, Glue, Informatica, Dataproc, Talend, Azure ADF). - Hands-on experience in cloud data services (AWS, Azure, or GCP). - Good understanding of data modeling, schemas, and data warehouse optimization techniques. - Knowledge of data governance and compliance frameworks. Additional Company Details (if available): The company is focused on Information Technology Data Engineering & Analytics Solutions. Note: The above job description is for a Senior Data Engineer role based in Bangalore / Hyderabad / Noida with a hybrid work model of 3 days in the office.,