CT&I - Software and Product Innovation - Data Engineering - Senior Associate

Role Overview: At PwC, as a data and analytics engineering professional, your main focus will be on utilizing advanced technologies and techniques to create robust data solutions for clients. Your role is crucial in converting raw data into actionable insights, which facilitate informed decision-making and drive business growth. In the field of data engineering at PwC, your responsibilities will include designing and constructing data infrastructure and systems to enable efficient data processing and analysis. You will be tasked with developing and implementing data pipelines, data integration, and data transformation solutions. Key Responsibilities: - Design, develop, and maintain robust, scalable ETL pipelines using technologies such as Apache Spark, Kafka, and other big data tools. - Create scalable and reliable data architectures, including Lakehouse, hybrid batch/streaming systems, Lambda, and Kappa architectures. - Demonstrate proficiency in Python, PySpark, Spark, and possess a solid understanding of design patterns like SOLID. - Ingest, process, and store structured, semi-structured, and unstructured data from various sources. - Utilize cloud platforms such as AWS, Azure, and GCP to set up data pipelines. - Optimize ETL processes to ensure scalability and efficiency. - Handle various file formats like JSON, CSV, Parquet, and Avro. - Deep knowledge of RDBMS, NoSQL databases, and CAP theorem principles. - Collaborate with data scientists, analysts, and stakeholders to understand data requirements and optimize data models. - Document data processes, architectures, and models comprehensively to facilitate cross-team understanding. - Implement and maintain CI/CD pipelines using tools like Docker, Kubernetes, and GitHub. - Ensure data quality, integrity, and security across all systems and processes. - Implement and monitor data governance best practices. - Stay updated with emerging data technologies and trends, identifying opportunities for innovation and improvement. - Knowledge of other Cloud Data/Integration/Orchestration Platforms like Snowflake, Databricks, and Azure Data Factory is advantageous. Qualification Required: - Minimum 4-7 years of experience in Programming Language (Python, Scala, Java), Apache Spark, ADF, Azure Databricks, Postgres, with familiarity in NoSQL. - BE / masters in design / B Design / B.Tech / HCI Certification (Preferred),