Senior Data Engineering Lead

As a Senior Data Engineer at our company, you will play a crucial role in designing, developing, and optimizing data pipelines, data lakes, and large-scale distributed systems. Your expertise in cloud platforms, ETL tools, data warehousing, and performance tuning will be essential for success in this role. You will work hands-on with technologies like Python, PySpark, SQL, and AWS, as well as big data platforms such as Databricks, Snowflake, BigQuery, and Delta Lake. **Key Responsibilities:** - Design & Develop Pipelines: Build scalable data pipelines for ingestion, transformation, and storage from multiple sources (databases, APIs, streaming). - Data Warehousing: Work with Snowflake, BigQuery, Delta Lake, and other lakehouse architectures to ensure performance, scalability, and cost optimization. - Cloud & ETL Tools: Leverage ETL/ELT frameworks such as Apache Airflow, Informatica, Talend, AWS Glue, Dataproc, or Azure ADF. - Programming & Scripting: Write clean, efficient code in Python, PySpark, and SQL for data manipulation and transformation. - Performance Tuning: Optimize pipelines for speed, cost efficiency, and minimal resource consumption. - DevOps & Infrastructure: Collaborate with DevOps teams for CI/CD integration, IaC (Terraform/CloudFormation), monitoring, and infrastructure provisioning. - Compliance & Security: Ensure pipelines meet data governance, audit, and security standards. - Collaboration: Work with architects, analysts, and business stakeholders to translate requirements into scalable solutions. - Mentorship: Guide and mentor junior engineers, ensuring adherence to best practices, coding standards, and certifications. **Qualifications Required:** - Strong expertise in Data Lakes, AWS, Python, PySpark, SQL. - Solid experience in distributed data processing frameworks. - Proficiency in ETL tools (Airflow, Glue, Informatica, Dataproc, Talend, Azure ADF). - Hands-on experience in cloud data services (AWS, Azure, or GCP). - Good understanding of data modeling, schemas, and data warehouse optimization techniques. - Knowledge of data governance and compliance frameworks. In addition to the above, you will collaborate with various teams, participate in the interview process consisting of 2 technical rounds and 1 client discussion, and utilize your skills in data, AWS, ETL, pipelines, SQL, and cloud data services to drive success in this role.,