Senior Technical Specialist

Bengaluru

Posted:15 hours ago

Job Description

Job Summary

The Databricks Developer will design, develop, test, and optimize scalable data-processing solutions using Apache Spark, PySpark, Delta Lake, and the Databricks platform. The role requires practical experience in reliable batch and change-data-capture pipelines, cloud integration, workload troubleshooting, and storage and processing optimization.

Key Responsibilities
  • Design, build, and maintain scalable Spark and PySpark data pipelines.
  • Develop Delta Lake solutions with effective schema handling, data quality controls, and read/write patterns.
  • Implement batch and CDC pipelines for incremental processing.
  • Optimize Spark jobs, storage layouts, and pipeline execution for performance, reliability, and cost.
  • Develop Databricks notebooks, workflows, jobs, and reusable components.
  • Integrate Databricks workloads with AWS services such as S3, EC2, ECS, and DynamoDB.
  • Troubleshoot pipeline failures, data-quality issues, performance bottlenecks, and production incidents.
  • Apply modular development, version control, testing, code review, deployment automation, and documentation practices.
  • Use REST APIs and platform automation where required.
Skill Requirements
  • Apache Spark knowledge from core concepts through distributed processing patterns.
  • Production-grade Python and PySpark development.
  • Practical Delta Lake and Delta table experience.
  • Storage, file-layout, Spark execution, and pipeline optimization.
  • Databricks notebooks, jobs, workflows, compute, and development practices.
  • Strong in Python data structures and algorithms.
  • AWS data-engineering services, particularly S3, with familiarity with EC2, ECS, and DynamoDB.
  • CDC and incremental data-processing solutions.
  • REST APIs for integration and automation.
  • Ability to develop solutions using Databricks Apps when required.
Must Have Skill
Databricks,PySpark,Python
Good to Have Skill
RESTAPI
Other Requirements

Good-to-Have Qualifications

  • Spark Structured Streaming fundamentals.
  • Exposure to Mosaic AI capabilities.
  • Relevant Databricks certification or preparation toward certification.

Expected Developer Profile

The preferred candidate is a hands-on developer who can translate requirements into maintainable Databricks solutions. The strongest emphasis is on Spark, PySpark, Delta Lake, pipeline and storage optimization, AWS integration, and CDC.

Skills Required

Data Processing Apache Spark Pyspark Cloud Integration Troubleshooting Spark Data Quality AWS Version Control Code Review Automation Rest Apis

Learn the skills this job asks for

Free courses, interview questions and practice for each skill.

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
HCLTech logo
HCLTech

Information Technology Services

New Delhi

RecommendedJobs for You