Job
Description
As a Senior Data and AI Engineer at TBO, you will play a crucial role in designing, developing, and maintaining data pipelines and ETL workflows using Apache Spark and AWS Glue. Your responsibilities will include building and optimizing large-scale data lake solutions leveraging HDFS, Apache Hudi, and AWS S3. Additionally, you will be involved in developing and managing data ingestion frameworks, transformations, and data modeling for analytical workloads. You will also design and develop AI/ML pipelines using pyspark, LLM, Langchain, RAG to solve business usecases. Furthermore, you will enable data discovery and querying using AWS Athena and other analytical tools, and implement and monitor data quality, security, and governance practices across the data platform. Collaboration with DevOps and Cloud teams to ensure data infrastructure is reliable, secure, and cost-efficient is an essential part of your role. **Key Responsibilities:** - Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and AWS Glue. - Build and optimize large-scale data lake solutions leveraging HDFS, Apache Hudi, and AWS S3. - Develop and manage data ingestion frameworks, transformations, and data modeling for analytical workloads. - Design & Develop AI/ML pipelines using pyspark, LLM, Langchain, RAG to solve business usecases. - Enable data discovery and querying using AWS Athena and other analytical tools. - Implement and monitor data quality, security, and governance practices across the data platform. - Work closely with DevOps and Cloud teams to ensure data infrastructure is reliable, secure, and cost-efficient. **Required Skills & Experience:** - 5-8 years of hands-on experience in Data Engineering or related roles. - Strong proficiency in Apache Spark (PySpark/Scala) and AWS Glue. - Solid understanding of HDFS, Data Lake architectures, and distributed data processing. - Hands-on experience with Apache Hudi for incremental data processing and time-travel capabilities. - Experience in AWS ecosystem - S3, Athena, Lambda, IAM, EMR, etc. - Strong SQL skills and experience with query optimization. - Good understanding of data modeling, ETL design patterns, and data lifecycle management. - Familiarity with version control (Git) and CI/CD processes for data pipelines. - Exposure to building an end-to-end data science solution is a plus.,
