Purpose of the Job
We are looking for a Senior Data Engineer who is equally strong in data modelling, data quality, SQL, and business understanding, as well as engineering skills such as Python, PySpark, CI/CD, performance optimization, and reliability.
The candidate will be responsible for designing and delivering production-ready data platforms, from gathering requirements to deployment and production support. The role also involves working directly with clients and sharing technical knowledge with the wider engineering team.
Technology Stack:
Key Responsibilities
- Own the end-to-end design and delivery of data pipelines and platforms, including requirements gathering, architecture, development, testing, deployment, and production support.
- Design data models, including dimensional modelling, SCD, and Medallion Architecture (Bronze/Silver/Gold), based on business requirements.
- Build and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark, and Azure Data Factory on Azure Databricks or Microsoft Fabric.
- Implement batch, incremental/CDC, and streaming data loads.
- Optimize Spark and SQL performance, ensuring scalability and cost efficiency.
- Implement data quality checks, monitoring, and alerting across pipelines.
- Apply data governance, lineage, and access control practices.
- Automate testing and deployment using Git and CI/CD while following engineering best practices.
- Work directly with client stakeholders to gather requirements, explain technical design decisions, and establish acceptance criteria.
- Integrate data from APIs, transactional systems, and external platforms.
- Investigate and resolve complex production issues and document their root causes.
- Maintain technical documentation and contribute to team knowledge-sharing through peer reviews and technical discussions.
Essential Qualifications
- Bachelor's degree in any discipline, preferably in Computer Science, Information Systems, or a related field.
- 58 years of relevant experience in Data Engineering.
Required Skills and Qualifications
Data Engineering Skills:
- Strong SQL knowledge.
- Experience in dimensional modelling and Medallion/Lakehouse architecture.
- Understanding of data quality practices.
- Ability to translate business requirements into effective data solutions.
Technical and Engineering Skills:
- Strong hands-on experience with Python and PySpark.
- Knowledge of Git, CI/CD, and automated testing.
- Experience in performance tuning of high-volume workloads.
- Hands-on experience with Azure Databricks or Microsoft Fabric.
- Experience with Azure Data Factory or an equivalent orchestration tool.
- Proven experience in developing batch and incremental/CDC pipelines using Delta Lake or similar technologies.
Client Management Skills:
- Experience in gathering client requirements.
- Strong stakeholder communication and collaboration skills.
- Ability to prepare clear technical documentation.
- Ability to demonstrate measurable improvements in runtime, cost, or data quality.
Good to Have
- Experience with streaming technologies such as Spark Structured Streaming and Kafka.
- Knowledge of governance and lineage tools such as Unity Catalog and Azure Purview.
- Understanding of Power BI and semantic-layer concepts.
- Microsoft certifications such as DP-203 or DP-600.
- Experience with Infrastructure as Code (Terraform).
Benefits and Additional Information
- Medical insurance coverage for employees.
- Provident Fund (PF) facility.
- Quarterly team parties and annual outings/trips.
- Regular leadership check-ins for professional growth and feedback.
- Recognition awards for high performance.
- Team engagement activities and events throughout the year.