Pyspark and DataBricks 5+years of expereince

Company Description

PTS Consulting is a prominent player in the IT services and consulting sector, with a strong focus on serving clients in South East Asia, the US, India, and Australia. The company provides end-to-end consulting services, bridging the gap between business vision and its practical implementation. Its core practice areas encompass management consulting, technology advisory, and comprehensive implementation services. Committed to continuous improvement and innovation, PTS Consulting leverages the latest knowledge and technologies to deliver value-driven solutions. The company’s expertise in Agile and waterfall methodologies ensures effective project management tailored to client needs.

Role Description: Senior Data Engineer (PySpark & Databricks)

Location: Bhubaneswar, Odisha (On-site)

Experience: 5+ Years and

Notice Period: Immediate joiner

Employment Type: Full-time

Role Overview:

We are seeking a seasoned Senior Data Engineer to join our growing data practice in Bhubaneswar. In this role, you will be the backbone of our data architecture, responsible for designing, building, and optimizing high-performance data pipelines.

You will leverage your deep expertise in PySpark and Databricks to transform raw data into actionable insights, ensuring our data infrastructure is scalable, reliable, and secure.

Key Responsibilities

 * Pipeline Development: Design and implement complex end-to-end ETL/ELT pipelines using PySpark and Databricks Delta Lake.

 * Architecture & Optimization: Optimize Spark jobs for performance and cost-efficiency. Implement best practices for data partitioning, caching, and shuffling.

 * Data Modeling: Build robust Medallion architectures (Bronze, Silver, Gold layers) to ensure data quality and accessibility.

 * Collaboration: Work closely with Data Scientists and Analysts to understand data requirements and provide production-ready datasets.

 * Automation: Implement CI/CD pipelines for data engineering workflows and automate monitoring/alerting systems.

 * Mentorship: Lead code reviews and provide technical guidance to junior engineers in the Bhubaneswar hub.

Required Skills & Qualifications

Technical Expertise

 * Databricks: Hands-on experience with Databricks Workflows, Unity Catalog, Delta Live Tables (DLT), and Photon engine.

 * PySpark: Expert-level proficiency in Python and Spark SQL for large-scale data processing.

 * Cloud Platforms: Strong experience with Azure (Azure Data Factory, ADLS Gen2) or AWS (S3, EMR).

 * Data Warehousing: Solid understanding of Lakehouse architecture and SQL performance tuning.

 * Version Control: Proficiency with Git and CI/CD tools (e.g., Azure DevOps, Jenkins, or GitHub Actions).

Professional Experience

 * Minimum of 5 years of professional experience in data engineering.

 * Proven track record of deploying production-grade data solutions in a cloud environment.

 * Strong analytical and problem-solving skills with a focus on "clean code" principles.

Preferred Qualifications

 * Databricks Certified Data Engineer Professional.

 * Experience with streaming technologies like Apache Kafka or Spark Structured Streaming.

 * Knowledge of Infrastructure as Code (Terraform or Bicep).

Why Join Us in Bhubaneswar?

 * Work on cutting-edge global projects while staying in the "Temple City."

 * Competitive salary and performance-based bonuses.

* Opportunity to be a founding member of our regional technical excellence center.

Please reach out us on geetha.murali@ptsconsulting.in for further details.

Thanks