Senior Digital Solutions Consultant

As an Integration Developer at Worley, you will play a crucial role in developing and maintaining data pipelines for ingesting and collecting data from various sources into a centralized data platform. Your responsibilities will include optimizing and troubleshooting AWS Glue jobs, utilizing Python and PySpark for data handling, collaborating with data architects on designing data models, and creating ETL processes using tools like Airflow, Python, and PySpark. Additionally, you will be involved in implementing monitoring solutions, managing databases, and working with Infrastructure as Code tools like Terraform and AWS CDK. Key Responsibilities: - Develop and implement data pipelines for collecting data from various sources - Maintain ETL jobs using AWS Glue for data processing and transformation - Optimize AWS Glue jobs for performance and reliability - Collaborate with data architects to design and implement data models - Create and maintain ETL processes using Airflow, Python, and PySpark - Implement monitoring solutions to track data pipeline performance - Manage and optimize SQL and NoSQL databases - Familiarity with Infrastructure as Code tools like Terraform and AWS CDK - Proficiency in event-driven and API-led data integrations - Experience with CI/CD pipelines such as Azure DevOps and AWS pipelines Qualifications Required: - Over 5 years of experience in integration projects - Proficiency in Python, PySpark, and SQL - Hands-on experience with AWS Glue, Airflow, Dynamo DB, Redshift, S3 buckets, and other AWS services - Experience with CI/CD pipelines and data testing practices - Proficient in web service development using Swagger, JSON, XML, SOAP, and REST - Degree in Computer Science or related fields - AWS Data Engineering certifications are desirable,