Data Engineer

Key Responsibilities:

• Develop, and deploy end-to-end data pipelines and solutions on Databricks,
• integrating with various data sources and systems.
• Collaborate with cross-functional teams to understand data, and deliver effective BI solutions.
• Implement data ingestion, transformation, and processing workflows using Spark(PySpark/Scala), SQL, and Databricks notebooks.
• Develop and maintain data models, ETL/ELT processes ensuring high performance, reliability, scalability and data quality.
• Build and maintain APIs and data services to support analytics, reporting, and application integration.
• Ensure data quality, integrity, and security across all stages of the data lifecycle.
• Monitor, troubleshoot, and optimize pipeline performance in a cloud-based environment.
• Integrate data from various sources, including APIs, relational and non-relational databases, IoT devices, and external data providers.
  Ensure adherence to data governance, security, and compliance policies.

Required Skills and Experience:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3-6 years of hands-on experience in data engineering, with a strong focus on Databricks and Apache Spark.
• Strong programming skills in Python/PySpark and/or Scala, with a deep understanding of Apache Spark.
• Experience with Azure Databricks.
• Strong SQL skills for data manipulation, analysis, and performance tuning.
• Strong understanding of data structures and algorithms, with the ability to apply them to optimise code and implement efficient solutions.
• Strong understanding of data architecture, data modelling, ETL/ELT processes, and data warehousing concepts.
• Experience building and maintaining ETL/ELT pipelines in production environments.
• Familiarity with Delta Lake, Unity Catalogue, or similar technologies.
  Experience working with structured and unstructured data, including JSON, Parquet, Avro, and time-series data.
• Familiarity with CI/CD pipelines and tools like Azure DevOps, version control (Git), and DevOps practices for data engineering.
• Excellent problem-solving skills, attention to detail, and ability to work independently or as part of a team.
  Strong communication skills to interact with technical and non-technical stakeholders.