Data Engineer

Job Description

5+ years of experience working in data warehousing systems 3+ strong hands-on programming expertise in Databricks landscape, including SparkSQL, Workflows for data processing and pipeline development 3+ strong hands-on data transformation/ETL skills using Spark SQL, Pyspark, Unity Catalog working in Databricks Medallion architecture 2+ yrs work experience in one of cloud platforms: Azure, AWS or GCP Experience working in using Git version control, and well versed with CI/CD best practices to automate the deployment and management of data pipelines and infrastructure Nice to have hands-on experience building data ingestion pipelines from ERP systems (Oracle Fusion preferably) to a Databricks environment, using Fivetran or any alternative data connectors Experience in a fast-paced, ever-changing and growing environment Understanding of metadata management, data lineage, and data glossaries is a plus Must have report development experience using PowerBI, SplashBI or any enterprise reporting tool. Roles & Responsibilities: Involved in design and development of enterprise data solutions in Databricks, from ideation to deployment, ensuring robustness and scalability. Work with the Data Architect to build and maintain robust and scalable data pipeline architectures on Databricks using PySpark and SQL Assemble and process large, complex ERP datasets to meet diverse functional and non-functional requirements. Involve in continuous optimization efforts, implementing testing and tooling techniques to enhance data solution quality. Focus on improving performance, reliability, and maintainability of data pipelines. Implement and maintain PySpark and databricks SQL workflows for querying and analyzing large datasets Involve in release management using Git and CI/CD practices Develop business reports using SplashBI reporting tool leveraging the data from Databricks gold layer Qualifications Bachelors degree in computer science, Engineering, Finance or equivalent experience. Good communication skills.