Databricks Consultant

About GyanSys Inc.

GyanSys Inc.

Bengaluru, Hyderabad, and Pune

Job Title:

Experience:

Location:

Immediate Joiners Preferred

Roles and Responsibilities Databricks Consultant (35 Years Experience)

1. Data Engineering & Pipeline Development

• Design, develop, and maintain scalable

  ETL/ELT pipelines

  on

  Databricks using Apache Spark

  .
• Proficient in

  Python, SQL and R.

• Implement data transformations using

  PySpark, SQL, Delta Lake

  , and

  Databricks notebooks

  .
• Build and optimize

  batch and streaming

  data pipelines based on business requirements.

2. Platform Implementation & Integration

• Assist in the setup, configuration, and administration of the

  Databricks workspace

  , clusters, and Unity Catalog.
• Integrate Databricks with various

  data sources

  like Azure Data Lake, AWS S3, Kafka, and relational databases.
• Support ingestion from source systems and delivery to analytics platforms or data warehouses.

3. Performance Tuning & Optimization

• Optimize Spark jobs for

  cost efficiency and runtime performance

  .
• Monitor and troubleshoot cluster performance, job failures, and data quality issues.
• Implement

  data caching, partitioning, and optimization techniques

  using Delta Lake.

4. Collaboration & Stakeholder Communication

• Work closely with data scientists, analysts, and business users to gather and translate requirements.
• Support analytics and ML teams by preparing clean, structured, and timely datasets.
• Participate in Agile ceremonies (scrum, sprint planning, retrospectives) as part of a cross-functional team.

5. Documentation & Best Practices

• Create and maintain

  technical documentation

  for pipelines, data flows, and system components.
• Follow industry and organizational best practices for

  code quality, version control, and CI/CD

  .

6. Governance, Quality, and Security

• Implement data validation, quality checks, and logging frameworks within Databricks pipelines.
• Support

  data security, compliance

  , and

  role-based access control (RBAC)

  practices via Unity Catalog or cloud-native IAM.
• Ensure auditability and traceability of all jobs and data movement.