Pyspark Developer

Stack Digital

3 - 6 years

4 - 8 Lacs

kolkata mumbai new delhi hyderabad pune chennai bengaluru

Posted:Just now| Platform:

Apply

Skills Required

advanced analytics github resource utilization machine learning design development jenkins deployment management testing sql

Work Mode

Work from Office

Job Type

Full Time

Job Description

Data Engineering and Processing:
Develop and manage data pipelines using PySpark on Databricks.
Implement ETL/ELT processes to process structured and unstructured data at scale.
Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.
Databricks Platform Expertise:
Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL)
Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.
Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.
Manage Delta Lake for ACID transactions and data versioning.
Develop notebooks and workflows for end-to-end data solutions.
Cloud Platforms and Deployment:
Deploy and manage Databricks on Azure (e.g., Azure Databricks).
Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.
Optimize resource utilization and troubleshoot performance issues on the Databricks platform.
CI/CD and Testing:
Build and maintain CI/CD pipelines for Databricks workflows using tools like Azure DevOps, GitHub Actions, or Jenkins.
Write unit and integration tests for PySpark code using frameworks like Pytest or unittest.

More Jobs at Stack Digital

Infrastructure Support Engineer-

Chennai, Pune, Delhi, Mumbai, Bengaluru, Hyderabad, Kolkata

2 - 5 yrs

INR 4 - 7 Lacs

Machine Learning Engineer

Kolkata, Mumbai, New Delhi, Hyderabad, Pune, Chennai, Bengaluru

3 - 6 yrs

INR 5 - 9 Lacs

Data Analyst/ BI

Kolkata, Mumbai, New Delhi, Hyderabad, Pune, Chennai, Bengaluru

2 - 6 yrs

INR 5 - 9 Lacs

QA Automation

Kolkata, Mumbai, New Delhi, Hyderabad, Pune, Chennai, Bengaluru

3 - 8 yrs

INR 6 - 10 Lacs

SRE-5811 Professional

Kolkata, Mumbai, New Delhi, Hyderabad, Pune, Chennai, Bengaluru

3 - 8 yrs

INR 11 - 15 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Machine Learning Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.