Databricks with Python

Databricks with Python Developer

Roles and Responsibilities:

• Big Data Solution Development:

  Design, develop, and implement scalable data processing pipelines and analytics solutions using

  PySpark

  within the

  Databricks

  platform.

• Data Ingestion & Transformation:

  Write efficient PySpark code to extract, transform, and load (ETL/ELT) large volumes of data from various sources into Databricks.

• Data Lake/Warehouse Management:

  Work with data lake and data warehouse concepts, ensuring data quality, consistency, and efficient storage within Databricks (e.g., Delta Lake).

• Performance Optimization:

  Optimize PySpark jobs and Databricks notebooks for performance, cost-efficiency, and scalability, addressing bottlenecks in data processing.

• Analysis & Insights:

  Perform complex data analysis using PySpark to uncover insights, build data models, and support business intelligence and machine learning initiatives.

• Troubleshooting:

  Perform in-depth troubleshooting, debugging, and issue resolution for PySpark jobs, Databricks environments, and data pipeline failures.

• Collaboration:

  Work closely with data engineers, data scientists, business analysts, and other stakeholders to understand data requirements and deliver robust solutions.

• Code Quality & Best Practices:

  Write clean, modular, and well-documented PySpark code. Participate in code reviews and adhere to best practices for big data development.

• Automation:

  Implement automation for data pipelines, job scheduling, and monitoring within the Databricks ecosystem.

Preferred Candidate Profile:

• PySpark Expertise:

  Strong hands-on development experience with

  PySpark

  for big data processing and analytics.

• Databricks Platform:

  Proven experience working with the

  Databricks platform

  , including Databricks notebooks, clusters, Delta Lake, and related services.

• SQL Proficiency:

  Excellent proficiency in

  SQL

  for data manipulation and querying.

• Python Programming:

  Strong programming skills in

  Python

  for data engineering tasks.

• Big Data Concepts:

  Solid understanding of big data concepts, distributed computing, and data warehousing principles.

• Cloud Platforms (Plus):

  Familiarity with cloud services from AWS, Azure, or GCP, particularly those related to data storage and processing, is a plus.

• Problem-Solving:

  Excellent analytical and problem-solving skills with a methodical approach to complex data challenges.

• Communication:

  Strong verbal and written communication skills to articulate technical solutions and collaborate effectively within a team.

• Education:

  Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field.