Data Engineer

Job Description

Design, build, and test end-to-end data pipelines for Data Ingestion, Integration, and Curation. Implement automation for data workflows and testing to enhance efficiency and reliability with data quality checks. Continuously optimize data pipelines to support diverse workloads and business requirements, improving performance over time. Create, maintain, and scale our cloud-based data platform ensuring high availability and scalability. Address and minimize technical debt to ensure a streamlined and efficient codebase. Develop reusable frameworks and components to streamline data engineering processes and boost development efficiency. Lead key data engineering projects from inception to successful delivery, taking ownership and ensuring completion with minimal oversight. Collaborate closely with analysts, business partners, and other stakeholders to understand data requirements and deliver high-quality solutions. Document data processes, workflows, and systems comprehensively to maintain clarity and transparency. What You Bring to the Table: Experience with data processing tools such as Snowflake, DBT, Databricks, Azure (ADF and Fabric), and GCP - BigQuery. Expertise in database optimization (partitioning, group and sort keys, indexes, query optimization). Hands-on coding experience with languages like Python, PySpark, and SQL to access, extract, manipulate, and summarize data. Programming and/or scripting experience: Python, PySpark. Experience in automation and testing of data workflows, preferably Azure ADF. Familiarity with a broad base of analytical methods like data modeling, variable-based transformation & summarization, and algorithmic development. Whats needed- Basic Qualifications: 6+ years of hands-on experience in designing solutions on the cloud computing platforms like Snowflake, Microsoft Azure or Google Cloud. 8+ years of experience with cloud-based databases, preferably Snowflake or Google BigQuery. 7+ years of experience developing scalable solutions using Python or PySpark. 8+ years of working experience in writing complex SQL’s. 8+ years of working in Agile environments. 5+ years of experience in CI/CD IMMEDIATE- SERVING NP until month end