Architect/Lead Data Engineer - AWS Glue

As a Data Engineer at our company, you will play a crucial role in designing, developing, and maintaining data pipeline architecture to ensure the efficient flow of data across the organization. Your responsibilities will include: - Designing robust and scalable data pipelines using AWS Glue, Apache Airflow, and other relevant technologies. - Integrating various data sources like SAP HANA, Kafka, and SQL databases for seamless data flow and processing. - Optimizing data pipelines for enhanced performance and reliability. You will also be responsible for: - Designing and implementing data transformation processes to clean, enrich, and structure data for analytical purposes. - Utilizing SQL and Python for data extraction, transformation, and loading (ETL) tasks. - Ensuring data quality and integrity through rigorous testing and validation processes. Collaboration and communication are key aspects of this role, as you will: - Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver tailored solutions. - Collaborate with cross-functional teams to implement DataOps practices and enhance data life cycle management. In addition, you will: - Monitor data pipeline performance and implement enhancements to improve efficiency and reduce latency. - Troubleshoot and resolve data-related issues promptly to minimize disruption. - Maintain comprehensive documentation of data pipelines, transformations, and processes. The qualifications needed for this role include: - Extensive experience with AWS Glue, Apache Airflow, Kafka, SQL, Python, SAP HANA, and DataOps tools. - Proficiency in SQL for querying and managing relational databases. - Strong analytical and problem-solving skills. - Excellent communication and collaboration abilities. - Detail-oriented with a commitment to data quality and accuracy. If you possess the required technical expertise and soft skills, and are enthusiastic about working with cutting-edge technologies in data engineering and DataOps, we look forward to having you join our dynamic team.,