Architect/Lead Data Engineer - AWS Glue

As a Data Engineer at our company, you will play a critical role in designing, developing, and maintaining data pipeline architecture to ensure the efficient flow of data across the organization. **Key Responsibilities:** - Develop robust and scalable data pipelines using AWS Glue, Apache Airflow, and other relevant technologies. - Integrate various data sources, including SAP HANA, Kafka, and SQL databases, for seamless data flow and processing. - Optimize data pipelines for performance and reliability. **Data Management And Transformation:** - Design and implement data transformation processes to clean, enrich, and structure data for analytical purposes. - Utilize SQL and Python for data extraction, transformation, and loading (ETL) tasks. - Ensure data quality and integrity through rigorous testing and validation processes. **Collaboration And Communication:** - Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver solutions. - Collaborate with cross-functional teams to implement DataOps practices and improve data life cycle management. **Monitoring And Optimization:** - Monitor data pipeline performance and implement improvements for enhanced efficiency. - Troubleshoot and resolve data-related issues to minimize disruption to data workflows. - Implement and manage monitoring and alerting systems for proactive issue identification and resolution. **Documentation And Best Practices:** - Maintain comprehensive documentation of data pipelines, transformations, and processes. - Adhere to best practices in data engineering, including code versioning, testing, and deployment procedures. - Stay updated with the latest industry trends and technologies in data engineering and DataOps. **Required Skills And Qualifications:** - Extensive experience with AWS Glue, Apache Airflow, Kafka, SQL, and Python. - Strong knowledge of SAP HANA and familiarity with DataOps tools and methodologies. - Advanced SQL skills and proficiency in Python for data tasks. - Ability to work independently and manage multiple projects simultaneously. **Preferred Skills:** - Knowledge of Snowflake for cloud-based data warehousing solutions. - Experience with AWS data services like Redshift, S3, and Athena. - Familiarity with big data technologies such as Hadoop, Spark, and Hive. *Note: No additional details of the company were provided in the job description.*,