Data Engineer - Azure / AWS Databricks

Position Summary:

This role is accountable for running day-to-day operations of the Data Platform in Azure / AWS

Databricks.The role involves designing and implementing data ingestion pipelines from multiple sources using Azure Databricks, ensuring seamless and efficient pipeline executions, and adhering to

security, regulatory, and audit control guidelines.

Key Responsibilities:

● Design and implement data ingestion pipelines from multiple sources using Azure

Databricks.

● Ensure data pipelines run smoothly and efficiently with minimal downtime.

● Develop scalable and reusable frameworks for ingesting large and complex datasets.

● Integrate end-to-end data pipelines, ensuring quality and consistency from source

systems to target repositories.

● Work with event-based and streaming technologies to ingest and process data in

real-time.

● Collaborate with other project team members to deliver additional components such as

API interfaces and search functionalities.

● Evaluate the performance and applicability of various tools against customer

requirements and provide recommendations.

● Provide technical advice to the team and assist in issue resolution, leveraging strong

Cloud and Databricks knowledge.

● Provide on-call, after-hours, and weekend support as needed to maintain platform

stability.

● Fulfil service requests related to the Data Analytics platform efficiently.

● Lead and drive optimisation and continuous improvement initiatives within the team.

● Conduct technical reviews of changes as part of release management, acting as a

gatekeeper for production deployments.

● Adhere to data security standards and implement required controls within the platform.

● Lead the design, development, and deployment of advanced data pipelines and analytical

workflows on the Databricks Lakehouse platform.

● Collaborate with data scientists, engineers, and business stakeholders to build and scale

end-to-end data solutions.

● Own architectural decisions to ensure alignment with data governance, security, and

compliance requirements.

● Mentor and guide a team of data engineers, providing technical leadership and

supporting career development.

● Implement CI/CD practices for data engineering pipelines using tools like Azure DevOps,

GitHub Actions, or Jenkins.

Qualifications and Experience:

● Bachelor’s degree in IT, Computer Science, Software Engineering, Business Analytics, or equivalent

● Minimum of 7+ years of experience in the data analytics field.

● Proven experience with Azure/AWS Databricks in building and optimising data pipelines,

● architectures, and datasets.

● Strong expertise in Scala or Python, PySpark, and SQL for data engineering tasks.

● Ability to troubleshoot and optimize complex queries on the Spark platform.

● Knowledge of structured and unstructured data design, modelling, access, and storage

techniques.

● Experience designing and deploying data applications on cloud platforms such as Azure

or AWS.

● Hands-on experience in performance tuning and optimising code running in Databricks environment

● Strong analytical and problem-solving skills, particularly within Big Data environments.

● Experience with Big Data management tools and technologies including Cloudera, Python,

Hive, Scala, Data Warehouse, Data Lake, AWS, Azure.

Technical and Professional Skills:

Must Have:

● Excellent communication skills with the ability to interact directly with customers.

● Azure/AWS Databricks.

● Python / Scala / Spark / PySpark.

● Strong SQL and RDBMS expertise.

● HIVE / HBase / Impala / Parquet.

● Sqoop, Kafka, Flume.

● Airflow.

● Jenkins or Bamboo.

● Github or Bitbucket.

● Nexus.

Good to Have:

● Relevant accredited certifications for Azure, AWS, Cloud Engineering, and/or Databricks.

● Knowledge of Delta Live Tables (DLT).