Lead Engineer - Data Engg & AI

As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines and ETL workflows for processing large-scale structured/unstructured data. Your expertise in AWS Data Services (S3, Workflows, Databricks, SQL), big data processing, real-time analytics, and cloud data integration is crucial for this role. Additionally, your experience in Team Leading will be valuable. Key Responsibilities: - Redesign optimized and scalable ETL using Spark, Python, SQL, UDF - Implement ETL/ELT Databricks workflows for structured data processing - Create Data quality checks using Unity Catalog - Drive daily status calls, sprint planning meetings - Ensure security, quality, and compliance of data pipelines - Collaborate with data architects and analysts to meet business requirements Qualifications Required: - At least 8 years of experience in data engineering - Minimum of 2 years working on AWS services - Hands-on experience with tools like S3, Databricks, or Workflows - Knowledge in Adverity, experience with ticketing tools like Asana or JIRA, and data analyzing skills are advantageous - Strong SQL and data processing skills (e.g., PySpark, Python) - Experience in data cataloging, lineage, and governance frameworks Your contribution to CI/CD integration, observability, and documentation, as well as your ability to quickly analyze issues and drive collaboration within the team, will be instrumental in achieving the goals of the organization.,