AWS Pyspark Data Engineer

Role Overview: As a Senior Data Engineer at DATAECONOMY, you will be responsible for leading the end-to-end development of complex models for compliance and supervision. Your deep expertise in cloud-based infrastructure, ETL pipeline development, and financial domains will be crucial in creating robust, scalable, and efficient solutions. Key Responsibilities: - Lead the development of advanced models using AWS services such as EMR, Glue, and Glue Notebooks. - Design, build, and optimize scalable cloud infrastructure solutions with a minimum of 5 years of experience. - Create, manage, and optimize ETL pipelines using PySpark for large-scale data processing. - Build and maintain CI/CD pipelines for deploying and maintaining cloud-based applications. - Perform detailed data analysis and deliver actionable insights to stakeholders. - Work closely with cross-functional teams to understand requirements, present solutions, and ensure alignment with business goals. - Operate effectively in agile or hybrid agile environments, delivering high-quality results within tight deadlines. - Enhance and expand existing frameworks and capabilities to support evolving business needs. - Create clear documentation and present technical solutions to both technical and non-technical audiences. Qualification Required: - 5+ years of experience with Python programming. - 5+ years of experience in cloud infrastructure, particularly AWS. - 3+ years of experience with PySpark, including usage with EMR or Glue Notebooks. - 3+ years of experience with Apache Airflow for workflow orchestration. - Solid experience with data analysis in fast-paced environments. - Strong understanding of capital markets, financial systems, or prior experience in the financial domain is a must. - Proficiency with cloud-native technologies and frameworks. - Familiarity with CI/CD practices and tools like Jenkins, GitLab CI/CD, or AWS CodePipeline. - Experience with notebooks (e.g., Jupyter, Glue Notebooks) for interactive development. - Excellent problem-solving skills and ability to handle complex technical challenges. - Strong communication and interpersonal skills for collaboration across teams and presenting solutions to diverse audiences. - Ability to thrive in a fast-paced, dynamic environment.,