Senior Data Engineer

Job Description

Your role is only the beginning! As a Senior Data Engineer, you will design, build, and maintain batch and real-time data pipelines on On-premises and cloud platforms to ingest data from the various source systems. Post ingestion, you will be responsible for integrating the data asset in the data model and making the data ready for consumption for the end-user use cases and downstream applications. What we seek to accomplish together: Enhance, optimize, and maintain existing data ingestion, transformation, and extraction pipelines and assets built for reporting and analytics on Cloud(Azure + Databricks) and Big Data(Cloudera ) platforms. Work with the Product Owner to understand the priorities and OKRs for the quarter and gather detailed requirements from the initiative owners or program sponsor as per the Epics planned to be delivered in the quarter. Build new and optimized data pipelines, and assets to meet the end-user requirements. The Data pipelines must adhere to all the architecture, design, and engineering principles. Design the data pipelines and assets to meet non-functional requirements (Security, reliability, performance, maintainability, scalability, and usability). Most importantly, they should be designed to be to keep the computing cost low on the Cloud. Data wrangling, Data profiling, and data analysis for new datasets ingested from source systems and derived/built from existing datasets with the on-premises and cloud-native tools Functionally understand the Data assets working with various SMEs and apply the transformation rules required to build the target data asset. Coordinate with other teams for planning, design, governance, engineering and release management of processes and ensure timely and accurate delivery of data and services. A little about you: Bachelors Degree in Math, Statistics, Finance, Economics, Computer Science, Information Technology, Management of Information Systems or equivalent Minimally 7 years of experience working in Data Engineering Minimally 2 years of experience leading a team of Data Engineers Experience in building fully automated end-to-end data pipelines using on-premises or cloud-based data platforms. Cloud experience Azure-based analytics/reporting pipeline Hands-on experience with scheduling tools like Airflow, Control M, etc. Hands-on delivery of solutions for Reporting and Analytics use cases. Hands-On with advanced SQL on Data Warehouse, Big Data and Data Bricks Experience in data profiling, source-target mappings, ETL development, SQL optimization, testing, and implementation. Experience working on Cloud DWs Knowledgeable in Big Data tools like Spark (python/Scala), Hive, Impala, Hue, and storage (e.g., HDFS, HBase) Knowledge of working with Azure Databricks. Knowledgeable in CICD processes Bitbucket/GitHub, Jenkins, Nexus etc. Knowledgeable in managing structured and unstructured data types like JSON, xml, Avro Track record of implementing databases and data access middleware and high-volume batch and (near) real-time processing