Data Engineer-II

As a Data Engineer-II at TMRW House of Brands in Bengaluru, Karnataka, your role involves creating, implementing, and operating robust and scalable data pipelines for business intelligence and machine learning. You will be responsible for developing and maintaining core data framework and key infrastructures. Additionally, you will create and support the ETL pipeline to ensure correct data flow from existing and new sources to the data warehouse. Your responsibilities also include designing the Data Warehouse and data modeling for efficient and cost-effective reporting. Collaboration with data analysts, data scientists, and other data consumers is essential to manage the data warehouse table structure and optimize it for reporting. You will also play a key role in defining and implementing Data Governance processes related to data discovery, lineage, access control, and quality assurance. Furthermore, you will perform code reviews and QA for data imported by various processes. Key Responsibilities: - Create, implement, and operate robust and scalable data pipelines for business intelligence and machine learning. - Develop and maintain core data framework and key infrastructures. - Create and support the ETL pipeline for correct data flow to the data warehouse. - Design Data Warehouse and data modeling for efficient reporting. - Collaborate with data analysts, data scientists, and data consumers to optimize data warehouse table structure. - Define and implement Data Governance processes for data discovery, lineage, access control, and quality assurance. - Perform code reviews and QA for data imported by various processes. Qualifications: - 3-5 years of experience in data engineering. - At least 3+ years of experience in data infrastructure on big data technologies like Hive, Spark, Pyspark (Batch and Streaming), Airflow, Redshift, and Delta Lake. - Experience in product-based companies or startups. - Strong understanding of data warehousing concepts and the data ecosystem. - Design/Architecture experience in AWS solutions. - Experience in building and managing data pipelines after deployment. - Familiarity with building data pipelines from business applications using APIs. - Previous experience with Databricks is advantageous. - Understanding of Dev Ops is preferable. - Working knowledge of BI Tools like Metabase and Power BI is a plus. - Experience in architecting systems for data access is highly beneficial.,