AWS Databricks : Senior Associate

PwC Acceleration Centers in India

4 years

0 Lacs

Bengaluru Karnataka India

Posted:3 weeks ago| Platform:

Apply

Skills Required

aws databricks pwc data analytics engineering design processing analysis integration apache spark etl architecture storage api optimization indexing strategies governance unity security controls migration python development orchestration airflow code git github writing sql communication teradata kafka jenkins automation terraform deployment

Work Mode

On-site

Job Type

Full Time

Job Description

At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data transformation solutions. Years of Experience: Candidates with 4+ years of experience in developing and delivering scalable big data pipelines using Apache Spark and Databricks on AWS. ` Position Requirements Must Have : Build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimize ETL/ELT processes for structured and unstructured data. Knowledge of Lakehouse architecture for efficient data storage, processing, and analytics. Orchestrating ETL/ELT Pipelines: Design and manage data workflows using Databricks Workflows, Jobs API. Worked with AWS Data Services (S3, Lambda, CloudWatch) for seamless integration. Performance Optimization: Optimize queries using pushdown capabilities and indexing strategies. Worked on data governance with Unity Catalog, security policies, and access controls. Monitor, troubleshoot, and improve Databricks jobs and clusters. Exposure to end-to-end implementation of migration projects to AWS Cloud AWS & Python Expertise with hands-on cloud development. Orchestration: Airflow Code Repositories: Git, GitHub. Strong in writing SQL Cloud Data Migration: Deep understanding of processes. Strong Analytical, Problem-Solving & Communication Skills. Good To Have Knowledge / Skills Experience in Teradata, DataStage , SSIS Knowledge of Databricks Delta Live Table. Knowledge of Delta Lake. Streaming: Kafka, Spark Streaming. CICD: Jenkins IaC & Automation: Terraform for Databricks deployment. Professional And Educational Background BE / B.Tech / MCA / M.Sc / M.E / M.Tech / MBA Show more Show less