Data Engineer

As a Data Engineer on Contract with 4-8 years of experience, you will play a crucial role in building and maintaining data systems on AWS. Your responsibilities will include designing and constructing efficient data pipelines using Spark, PySpark, or Scala, managing complex data processes with Airflow, and cleaning, transforming, and preparing data for analysis. You will utilize Python for data tasks, automation, and tool development, while collaborating with the Analytics team to understand their data requirements and provide appropriate solutions. In this role, you will work with various AWS services such as S3, Redshift, EMR, Glue, and Athena to manage the data infrastructure effectively. Additionally, you will be involved in developing and maintaining a Node.js backend for data services, utilizing Typescript, and managing settings for data tools using YAML. Setting up automated deployment processes through GitHub Actions, monitoring and resolving issues in data pipelines, implementing data accuracy checks, and designing data warehouses and data lakes will also be part of your responsibilities. As a qualified candidate, you should possess a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with strong skills in Spark, Scala, and Airflow. Proficiency in Python, experience with AWS cloud services, familiarity with Node.js and Typescript, and hands-on experience with GitHub Actions for automated deployment are essential requirements. A good understanding of data warehousing concepts, strong database skills, proficiency in SQL, experience with stream processing using Kafka, and excellent problem-solving, analytical, and communication skills are also expected. Additionally, guiding and mentoring junior data engineers and contributing to the development of data management rules and procedures will be part of your role.,