Sr. Data Engineer (C# and Python)

We are seeking a highly skilled Senior Data Engineer 2 with extensive experience in analysing existing data/databases, designing, building, and optimizing high-volume data pipelines. The ideal candidate will have strong expertise in Python, Databases, Databricks on Azure Cloud services, DevOps, and CI/CD tools, along with a solid understanding of AI/ML techniques and big data processing frameworks like Apache Spark and PySpark. 

Responsibilities

• Adhere to coding and Numerator technology standards  
• Build suitable automation test suites within Azure DevOps  
• Maintain and update automation test suites as required  
• Carry out manual testing, load testing, exploratory testing as required  
• Perform Technical Analysis and work closely with Business Analysts and Senior Data Developers to consistently deliver sprint goals  
• Assist in estimation of sprint-by-sprint stories and tasks  
• Pro-actively take a responsible approach to product delivery 

Requirements

• 7-10 years of experience in data engineering roles, handling large databases 
• Good C# and Python skills 
• Experience working with Microsoft Azure Cloud  
• Experience in Agile methodologies (Scrum/Kanban) 
• Experience with Apache Spark, PySpark, Databricks  
• Experience working with Devops pipeline, preferably Azure DevOps 

Preferred Qualifications

• Bachelor's or master's degree in computer science, Information Technology, Data Science, or a related field. 
• Experience working in a technical development/support focused role  
• Knowledge/experience in AI ML techniques 
• Knowledge/experience in Visual Basic 6 
• Certification in relevant Data Engineering discipline or related fields.