Azure Databricks writing code in Python and SQL with PySpark

As a Data Engineer Grade 5 at our company, your role involves working on solutions against delta lakes and relational databases. You will primarily focus on Azure Databricks, utilizing Python and SQL with PySpark for coding. Your responsibilities will include analysis, design, implementation, and testing, with a strong emphasis on automated testing. Additionally, you will be extending the functionality of the consistency check engine for sub-ledger data and other tenants, dealing with large datasets containing billions of rows and hundreds of columns, requiring deep dive into Spark performance optimization. Key Responsibilities: - Build solutions against delta lakes and relational databases - Utilize Azure Databricks for coding in Python and SQL with PySpark - Perform analysis, design, implementation, and testing - Extend the functionality of the consistency check engine for sub-ledger data and other tenants - Optimize Spark performance for large datasets Qualifications Required: - Bachelor's or master's degree in computer science or electrical engineering - Expert software engineering skills with proven experience in developing complex software systems - Strong core Python software engineering experience - Significant experience with projects using relational data models and SQL - Spark experience is essential - Good work ethic, self-starter mentality, and proactive approach to problem-solving - Strong analytical skills to understand complexity and break it down into achievable steps - Proficiency in English (written and verbal) If you are a proactive and experienced software engineer with a background in building solutions against delta lakes and relational databases, we encourage you to apply for this position by sharing your CV at krishna.bhagdeo@infoorigin.com. Don't miss this opportunity! APPLY NOW!,