As a Consultants in our Consulting team, you ll build and nurture positive working relationships with teams and clients with the intention to exceed client expectations. You ll:
Design, develop, and optimize ETL pipelines using PySpark and Databricks to process large-scale data on the Azure cloud platform.
Implement data ingestion processes from various data sources into Azure Data Lake and Azure SQL Data Warehouse.
Develop and maintain data models, data schemas, and data transformation logic tailored for Azure.
Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
Ensure data quality and integrity through robust testing, validation, and monitoring procedures.
Optimize and tune PySpark jobs for performance and scalability within the Azure and Databricks environments.
Implement data governance and security best practices in Azure.
Monitor and troubleshoot data pipelines to ensure timely and reliable data delivery.
Document data engineering processes, workflows, and best practices specific to Azure and Databricks.
Requirements:
Bachelors or Masters degree in Computer Science, Engineering, or a related field.
Proven experience as a Data Engineer with a strong focus on PySpark and Databricks.
Proficiency in Python and PySpark for data processing and analysis.
Strong experience with Azure data services, including Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, and Azure Databricks.
Strong SQL skills and experience with relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
Experience with big data technologies such as Hadoop, Spark, Hive, and Kafka.
Strong understanding of data architecture, data modeling, and data integration techniques.
Familiarity with Azure DevOps, version control systems (e.g., Git), and CI/CD pipelines.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills.
Preferred Qualifications:
Experience with Delta Lake on Azure Databricks.
Knowledge of data visualization tools (e.g., Power BI, Tableau).
Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
Understanding of machine learning concepts and experience working with data scientists.
Your role as a leader
At Deloitte India, we believe in the importance of leadership at all levels. We expect our people to embrace and live our purpose by challenging themselves to identify issues that are most important for our clients, our people, and for society and make an impact that matters.
In addition to living our purpose, Consultant across our organization:
Develop high-performing people and teams through challenging and meaningful opportunities
Deliver exceptional client service; maximize results and drive high performance from people while fostering collaboration across businesses and borders
Influence clients, teams, and individuals positively, leading by example and establishing confident relationships with increasingly senior people
Understand key objectives for clients and Deloitte; align people to objectives and set priorities and direction.
Acts as a role model, embracing and living our purpose and values, and recognizing others for the impact they make