Job
Description
At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data transformation solutions. Years of Experience: Candidates with 8+ years of experience in architecting and delivering scalable big data pipelines using Apache Spark and Databricks on AWS. Position Requirements Must Have : Design, build, and maintain scalable data pipelines using Databricks and Apache Spark. Good knowledge on Medallion Architecture in Databricks Lakehouse Develop and optimize ETL/ELT processes for structured and unstructured data. Implement Lakehouse architecture for efficient data storage, processing, and analytics. Orchestrating ETL/ELT Pipelines: Design and manage data workflows using Databricks Workflows, Jobs API. Work with AWS Data Services (S3, Lambda, CloudWatch) for seamless integration. Performance Optimization: Optimize queries using pushdown capabilities and indexing strategies. Implement data governance with Unity Catalog, security policies, and access controls. Collaborate with data scientists, analysts, and engineers to enable advanced analytics. Monitor, troubleshoot, and improve Databricks jobs and clusters. Strong expertise in end-to-end implementation of migration projects to AWS Cloud Should be aware of Data Management concepts and Data Modelling AWS & Python Expertise with hands-on cloud development. Spark Performance Tuning: Core, SQL, and Streaming. Orchestration: Airflow Code Repositories: Git, GitHub. Strong in writing SQL Cloud Data Migration: Deep understanding of processes. Strong Analytical, Problem-Solving & Communication Skills. Good To Have Knowledge / Skills Experience in Teradata, DataStage , SSIS, Mainframe(Cobol, JCL, Zeke Scheduler) Knowledge on Lakehouse Federation Knowledge of Delta Lake. Knowledge of Databricks Delta Live Table. Streaming: Kafka, Spark Streaming. CICD : Jenkins IaC & Automation: Terraform for Databricks deployment. Knowlege on integrating 3party APIs to Databricks. Knowledge of Transport & Mobility domain. Professional And Educational Background BE / B.Tech / MCA / M.Sc / M.E / M.Tech / MBA Show more Show less
