Analyst/Snr Analyst – Level 3
A Pyspark and Databricks Developer with a good understanding of the entire ETL/Azure lifecycle with a background of data projects.
Work you’ll do
1. Design and Build Data Pipelines
- Develop robust, scalable ETL/ELT pipelines using Databricks and Apache Spark.
- Integrate data from on-premises databases, cloud storage, APIs, and third-party sources into the enterprise data lake.
- Automate both batch and real-time data ingestion to support enterprise reporting and advanced analytics (with specific experience in SAP HANA integration being highly valuable).
- Enhance pipeline designs by leveraging Azure Data Factory (ADF), diverse connectors, and core cloud services to create seamless, end-to-end data flows.
2. Collaboration & Migration
- Collaborate with onshore teams, architects, fellow developers, and business analysts to align new data flows with overall enterprise data warehouse (EDW) migration strategies.
- Support the migration of legacy and on-premises systems (especially SAP HANA) to cloud-native services, including the use of Delta Lake on Databricks.
- Optimize data modeling for analytical workloads, including the development of star/snowflake schemas and dimensional models.
3. Data Transformation & Preparation
- Cleanse, transform, and join diverse datasets for analytics and machine learning use cases.
- Use PySpark (mandatory) and, optionally, Scala within Databricks notebooks and jobs for efficient data processing.
- Ensure high data quality through comprehensive validation, exception handling, and logging practices.
4. Performance Tuning & Cost Optimization
- Optimize Spark jobs for efficiency, cost, and scalability (utilizing partitioning, caching, and dynamic resource allocation).
- Monitor Databricks workloads and resolve bottlenecks affecting compute, storage, or data skew.
5. Data Governance & Security
- Assist in enforcing security controls such as access management, data masking, and encryption.
- Support enterprise-wide data governance policies, focusing on metadata management, data lineage, and auditability.
6. Advanced SQL Skillset
- Build, optimize, and maintain SQL queries for ETL processes within Databricks.
- Create and manage data models and tables with Databricks SQL or Spark SQL.
- Write and automate high-performance SQL scripts for joining, filtering, aggregating, and analyzing large datasets.
- Document SQL logic and data flows; convert and optimize SAP HANA SQL code to Databricks-compatible SQL as part of migration.
- Collaborate with analysts and end-users to deliver accurate reporting and dashboarding.
7. Automation & DevOps Integration
- Develop reusable components and orchestrate workflows; integrate pipelines with DevOps and CI/CD practices.
- Utilize ADO (Azure DevOps) for version control and automated deployment of data assets and Databricks notebooks.
8. Documentation & Best Practices
- Document all pipeline logic, ETL job definitions, and key system processes with clarity and thoroughness.
- Champion coding standards, reusable code design, and ongoing knowledge sharing within the team.
9. SAFe Agile Teamwork
- Work closely with business analysts, product owners, and data stewards to understand evolving requirements and build jobs accordingly.
- Provide transparent updates and proactively escalate technical challenges in alignment with SAFe Agile processes.
The team
Solutions Delivery-Canada is an integral part of the Information Technology Services group. The principle focus of this organization is the development and maintenance of technology solutions that e-enable the delivery of Function and Marketplace Services and Management Information Systems.
Solutions Delivery Canada develops and maintains solutions built on varied technologies like Siebel, PeopleSoft Microsoft technologies and Lotus Notes. Solutions Delivery Canada has various groups which provide the best of the breed solutions to the clients by following a streamlined system development methodology. Solutions Delivery Canada comprises of groups like Usability, Application Architecture, Development and Quality Assurance and Performance.
Qualifications
Required:
- Computer Science University degree or equivalent work experience
- At least 1.5-2.5 years for Aanalyst and 4-5 years of total exp
- Experience in Databricks, Pyspark and ADF etc.
- Experience in Databricks end-to-end implementation.
- Expert in developing solutions and delivering the high-quality projects using Pyspark and Databricks
- Proficiency with Azure Data Factory, Azure DevOps, and cloud-native data services.
- Experience integrating and migrating data from SAP HANA to cloud platforms (Good to have)
- Advanced SQL skills for data modeling and migration.
- Experience implementing data security and governance policies.
- Familiarity with DevOps, CI/CD, and infrastructure-as-code concepts
- Expert on data integration tools, such as Data Stage is good to have
- Knowledge of basic Python programming - Preferable
- Knowledge of Cloud based ETL services
- Excellent organizational and communication skills
- Must have strong interpersonal skills, and the ability to effectively work with others in teams
- Strong problem-solving skills in identifying issues and resolution
- Adeptness at learning new technology and tools
- The ability to effectively manage multiple assignments and responsibilities in a fast-paced environment
- Strong commitment to professional client service excellence
- Should be a strong individual contributor
Work Location:
Hyderabad
How you’ll grow
At Deloitte, we’ve invested a great deal to create a rich environment in which our professionals can grow. We want all our people to develop in their own way, playing to their own strengths as they hone their leadership skills. And, as a part of our efforts, we provide our professionals with a variety of learning and networking opportunities— including exposure to leaders, sponsors, coaches, and challenging assignments—to help accelerate their careers along the way. No two people learn in exactly the same way. So, we provide a range of resources including live classrooms, team-based learning, and eLearning. DU: The Leadership Center in India, our state-of-the-art, worldclass learning Center in the Hyderabad offices is an extension of the Deloitte University (DU) in Westlake, Texas, and represents a tangible symbol of our commitment to our people’s growth and development. Explore DU: The Leadership Center in India
Benefits
At Deloitte, we know that great people make a great organization. We value our people and offer employees a broad range of benefits. Learn more about what working at Deloitte can mean for you.
Deloitte’s culture
Our positive and supportive culture encourages our people to do their best work every day. We celebrate individuals by recognizing their uniqueness and offering them the flexibility to make daily choices that can help them to be healthy, centered, confident, and aware. We offer well-being programs and are continuously looking for new ways to maintain a culture that is inclusive, invites authenticity, leverages our diversity, and where our people excel and lead healthy, happy lives. Learn more about Life at Deloitte.
Corporate citizenship
Deloitte is led by a purpose: to make an impact that matters. This purpose defines who we are and extends to relationships with our clients, our people and our communities. We believe that business has the power to inspire and transform. We focus on education, giving, skill-based volunteerism, and leadership to help drive positive social impact in our communities. Learn more about Deloitte’s impact on the world.
Benefits to help you thrive
At Deloitte, we know that great people make a great organization. Our comprehensive rewards program helps us deliver a distinctly Deloitte experience that helps that empowers our professionals to thrive mentally, physically, and financially—and live their purpose. To support our professionals and their loved ones, we offer a broad range of benefits. Eligibility requirements may be based on role, tenure, type of employment and/ or other criteria. Learn more about what working at Deloitte can mean for you.
