Senior Software Engineer

Key responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines and ETL processes to extract, transform, and load data from various sources into our data lake or warehouse
• Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements, define data models, and implement solutions that meet business needs
• Ensure the security, integrity, and quality of data throughout the data lifecycle, implementing best practices for data governance, encryption, and access control
• Develop and maintain data infrastructure components such as data warehouses, data lakes, and data processing frameworks, leveraging cloud services (e g , AWS, Azure, GCP) and containerization technologies (e g , Docker, Kubernetes)
• Implement monitoring, logging, and alerting mechanisms to ensure the reliability and availability of data pipelines and systems, and to proactively identify and address issues
• Work closely with stakeholders to understand business requirements, prioritize tasks, and deliver solutions in a timely manner within an Agile working environment
• Collaborate with the risk, security and compliance teams to ensure adherence to regulatory requirements (e g , GDPR, PCI DSS) and industry standards related to data privacy and security
• Stay updated on emerging technologies, tools, and best practices in the field of data engineering, and propose innovative solutions to improve efficiency, performance, and scalability
• Mentor and coach junior engineers, fostering a culture of continuous learning and professional development within the team
• Participate in code reviews, design discussions, and other Agile ceremonies to promote collaboration, transparency, and continuous improvement

Qualifications

Required skills/ experience:

• Experience of the full development lifecycle
• Strong communication skills with the ability to explain solutions to technical and non technical audiences
• Write clean, scalable and re-usable code that implements SOLID principles, common
• design patterns where applicable and adheres to published coding standards
• Excellent attention to detail, ability to analyse, investigate and compare large data sets when required
• 3 or more years of programming using Scala
• 2 or more years of programming using Python
• Some experience of using Terraform to provision and deploy cloud services and components
• Experience of developing on Apache Spark
• Experience of developing with AWS cloud services including (but not limited to) AWS Glue, S3, Step Functions, Lambdas, EventBridge and SQS
• BDD / TDD experienceJenkins CI / CD experience
• Application Lifecycle Management Tools - BitBucket & Jira
• Performing Pull Request reviews
• Understanding of Agile methodologiesAutomated Testing Tools

Advantageous experience:

• Mentoring or coaching junior engineersCloud Solution Architecture
• Document databases
• Relational Databases
• Experience with Container technologies (e g Kubernetes)
• Would consider alternative skills and experience:
• Java (rather than Scala)
• Google Cloud or Microsoft Azure (rather than AWS)
• Azure Pipelines or TeamCity (rather than Jenkins)
• Github (rather than BitBucket)
• Azure DevOps (rather than Jira)
• CloudFormation (rather than Terraform)