Databricks Engineer- Technical lead

Locations:

Pune, Maharashtra, India

Minimum Experience:

8

Maximum Experience:

12

Skill to Evaluate:

azure/aws,databricks,SQL,PYSPARK, Team Handling

Job Title: Technical Lead
Company Name: NCSI

This role is accountable for running day to day operations of Data Platform in Azure / AWS Databricks NCS. Data Engineer is accountable for ongoing Development, Enhancement support and maintenance data availability and data quality, performance enhancement and stability of the system.

1. Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks

2. Ensure data pipelines run smoothly and efficiently

3. Adherence to security, regulatory and audit control guidelines

4. Drive optimization, continuous improvement and efficiency

1

Designing and implementing data ingestion pipelines from multiple sources using Azure

Databricks

2 Ensure data pipelines run smoothly and efficiently

3

Developing scalable and re-usable frameworks for ingesting of data sets

4 Integrating the end to end data pipeline - to take data from source systems to target data

repositories ensuring the quality and consistency of data is maintained at all times

5 Working with event based / streaming technologies to ingest and process data

6 Working with other members of the project team to support delivery of additional project

components (API interfaces, Search)

7 Evaluating the performance and applicability of multiple tools against customer requirements

8 Technically competent in Cloud and Databricks to provide technical advice to the team and

to be involved in issue resolution

9 Provide on-call support and afterhours/weekend support as needed

10 Fulfill Service Requests related to Data Analytics platform

11 Lead and drive optimization and continuous improvement initiatives

12 Play gate-keeping role and conduct technical review of the changes as part of release

management

13 Understand various data security standards and adhere to the required data security controls

in the platform

14 Lead the design, development, and deployment of advanced data pipelines and analytical

workflows on the Databricks Lakehouse platform.

15 Collaborate with data scientists, engineers, and business stakeholders to build and scale

end-to-end data solutions.

16 Own architectural decisions and ensure adherence to data governance, security, and

compliance requirements.

17 Mentor a team of data engineers, providing technical guidance and career development.

Implement CI/CD practices for data engineering pipelines using tools like Azure DevOps,

GitHub Actions, or Jenkins.

Bachelors degree in IT, Computer

Science, Software Engineering,

Business Analytics or equivalent.

• Relevant accredited

certifications for Azure, AWS,

Cloud Engineering and/or

Databricks

Work Experience

• Minimum seven plus years of

experience in data analytics field

• Experience with Azure/AWS

Databricks

• Experience in building and optimizing

data pipelines, architectures and data

sets

• Excellent experience in Scala or

Python, PySpark and SQL

• Ability to troubleshoot and optimize

complex queries on the Spark

platform

• Knowledgeable on structured and

unstructured data design / modelling,

data access and data storage

techniques

• Expertise in designing and deploying

data applications on cloud solutions,

such as Azure or AWS

• Hands on experience in performance

tuning and optimizing code running in

Databricks environment

• Demonstrated analytical and

problem-solving skills particularly

those that apply to a big data

environment