Sr. Software Engineer

Enhance, optimize and maintain existing data ingestion, transformation and extraction pipelines and assets built for reporting and analytics on Cloud (GCP + BigQuery + Confluent Cloud) and Big Data(Cloudera and Confluent Platform) platforms.
Work with the Product Owner to understand the priorities and OKRs for the quarter and gather detailed requirements from the initiative owners or program sponsor as per the Epics planned to be delivered in the quarter. Build new and optimized data pipelines, assets to meet the end user requirements. The Data pipelines must adhere to all the architecture, design and engineering principles. Design the data pipelines and assets to meet non functional requirements (Security, reliability, performance, maintainability, scalability, and usability). Most importantly, they should be designed to be to keep the compute cost low on Cloud. Data wrangling, Data profiling and data analysis for new datasets ingested from source systems and derived/built from existing datasets with the on premises and cloud native tools Functionally understand the Data assets working with various SMEs and apply the transformation rules required to build the target data asset. Coordinate with other teams for planning, design, governance, engineering and release management of processes and ensure timely and accurate delivery of data and services. Build the new data asset and data pipeline as per the downstream requirement. e.g. The Dataset for Tableau will be different to the Dataset built for TM1 cubes. Document the low level design, source to column mapping, Test cases, production release implementation plan, and Operation support Manual. Schedule the data pipeline using the existing tools like Control M with the correct upstream dependencies and meeting the SLA requirements. Provide warranty support to Operations post production release and update the documentation subsequently. Collaborate with multiple business and IT teams to deliver the final outcome.