22907 Pyspark Jobs - Page 4

**Urgent Hiring for the Role of Data Engineer** Role: Data Engineer Duration: 6 month Contract to Hire Location: Chennai, Hyderabad, Pune, and Delhi Work timing: IST Hours Job Description: Required Qualifications 5-10 years of overall IT experience, including a minimum of 3 years in big data engineering on Microsoft Azure. Expert-level SQL skills with the ability to write, optimize, and troubleshoot complex queries in Azure SQL, Synapse, or similar cloud data platforms. Strong knowledge of Spark architecture and core APIs (RDD, DataFrame, Dataset). Deep hands-on experience with the Databricks ecosystem, including Notebooks, Workflows, Unity Catalog, SQL Warehouse, Serverless compute, and oth...

Role & responsibilities Key Responsibilities: Design and implement scalable data pipelines using Databricks aligned with Medallion Architecture (Bronze, Silver, Gold layers) . Apply dimensional data modeling including Star, Snowflake, Galaxy schema etc. to develop business reports, dashboards, and advanced analytics. Collaborate with cross-functional teams to understand business requirements and translate them into technical data solutions. Ensure data quality, consistency, and reliability across all layers of the data architecture. Support Power BI and other analytics teams with well-structured data models and efficient query performance. Monitor, troubleshoot, and optimize Databricks jobs ...

Who We Are At Sonatype, we help organizations build better, more secure software by enabling them to understand and control their software supply chains. Our products are trusted by thousands of engineering teams globally, providing critical insights into dependency health, license risk, and software security. We’re passionate about empowering developers—and we back it with data. The Opportunity We’re looking for a Data Engineer to join our growing Data Platform team. You will design and build data-driven microservices and workflows using Java, Python, and Spring Boot, implement data workflows, and deploy everything through CI/CD pipelines into AWS ECS/Fargate. You’ll also ensure services ar...

Job Title: Data Engineer Location: Hyderabad (Hybrid) Experience: 6+ Years (STRICTLY) Employment Type: Permanent Notice Period: Immediate Joiners Only Mode of Interview: Face to Face at client location, Hyderabad About the Company Our client is a trusted global innovator of IT and business services, present in 50+ countries. They specialize in digital & IT modernization, consulting, managed services, and industry-specific solutions. With a commitment to long-term success, they empower clients and society to move confidently into the digital future. About the Role We are seeking a highly skilled Senior Data Engineer with 6+ years of experience to design, build, and optimize large-scale data p...

Job Title: Azure Data Engineer Location: On-Site Employment Type: Full-Time Work About The Role We are seeking a motivated Mid-level Data Engineer with hands-on experience in Databricks to join our team. The ideal candidate will have a strong foundation in data engineering principles and a passion for building scalable data solutions. This role involves working with modern data platforms to design, develop, and optimize data pipelines that support analytics and business intelligence. Key Responsibilities Design, develop, and maintain ETL pipelines using Databricks. Perform environment setup, establish data connections, and conduct testing within Databricks. Collaborate with data scientists, ...

Experience: 6+ Years Domain: Data Engineering / Big Data About the Role: We are seeking a Senior Data Engineer to drive the development of our next-generation Data Lakehouse architecture. You will be responsible for designing, building, and optimizing massive-scale, low-latency data pipelines that support real-time analytics and Machine Learning applications. Key Responsibilities: - Design and build highly optimized, production-grade ETL/ELT pipelines using Apache Spark (PySpark/Scala) to process petabytes of data. - Architect and manage the Data Lakehouse using open-source technologies like Delta Lake or Apache Hudi for ACID transactions and data quality. - Integrate and process real-time d...

Data Engineering concepts (Data Warehouse (DW), Data Lake, OLTP vs OLAP, etc.); API Development (Python); Docker & Kubernetes; Airflow; AWS project;Strong communication skills

Duties & Responsibilities Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis Translate business requirements into well-designed, documented, and reusable data solutions Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi,Databricks etc.) Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability Document pipelines, workflows, and design decisions for knowledge s...

We are seeking a highly skilled and motivated Big Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience with Hadoop ecosystem, Apache Spark, and programming expertise in Python (PySpark), Scala, and Java. You will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions to support analytics and business intelligence initiatives.

Experience: 6 to 10 years of hands-on experience in the data analytics space as a data engineer. Familiar with ETL, DQ, DM, and reject and recycling concepts. A significant portion of this experience should involve building data analytics solutions in a big data environment using Hadoop clusters or cloud environment. Technical Skills: Extensive experience in building data pipelines using Spark, particularly PySpark. Candidates must have a minimum of 3 years of hands-on experience in coding with PySpark applications using RRDs, DataFrames & datasets and NOT Spark SQLs. Candide should has developed numerous spark application for various use cases of processing large volumes of data, used perfo...

About Position: We are looking for Lead Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Azure Data Lead Engineer Location: Pune/ Bangalore Experience: 8 to 10 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tool...

About Position: We are looking for Sr Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Sr. Azure Data Engineer Location: Pune/Bangalore Experience: 5 to 8 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tools to ...

Key Responsibilities: Technical Strategy & Roadmap: Define and execute comprehensive ML/AI strategy aligned with business objectives, including LLM infrastructure, retrieval pipelines, and generative workflows Architecture Oversight: Lead system architecture decisions for scalable ML platforms,ensuring robust model deployment and operational excellence Cross-functional Collaboration: Partner closely with product management, engineering, and business stakeholders to translate requirements into technical solutions Model Development: Actively participate in designing, training, and optimizing machine learning models, maintaining technical currency with 20-30% hands-on contribution Code Reviews ...

About the Role We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques. The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work clos...

Design, implement, and maintain data pipelines for processing large datasets, ensuring data availability, quality, and efficiency for machine learning model training and inference. Collaborate with data scientists to streamline the deployment of machine learning models, ensuring scalability, performance, and reliability in production environments. Develop and optimize ETL (Extract, Transform, Load) processes, ensuring data flow from various sources into structured data storage systems. Ensure effective model monitoring, versioning, and logging to track performance and metrics in a production setting. Collaborate with cross-functional teams to improve data architectures and facilitate the con...

Job Title: Data Engineer Azure Fabric, PySpark & Migration Specialist Position Summary We are seeking a Data Engineer with strong expertise in Azure Fabric, PySpark, and data pipeline orchestration to lead the migration of existing tools and workflows from Azure Synapse and SQL Server stored procedures into a modern data platform built on Microsoft Fabric. A deep understanding of distributed data processing, Apache Airflow, and cloud-native data engineering is essential. This role will be pivotal in transforming our data infrastructure, with hands-on responsibilities in pipeline development, data modeling, and performance optimization using tools such as PySpark, Delta Lake, and OneLake. Key...

We are seeking a highly skilled and motivated Big Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience with Hadoop ecosystem, Apache Spark, and programming expertise in Python (PySpark), Scala, and Java. You will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions to support analytics and business intelligence initiatives.

5+ years of experience in Oracle, PL/SQL Develop PL/SQL Packages, Stored Procedures, Functions, Triggers, SQL statements Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes Developing scripts for reporting, data manipulation and automation of daily tasks Perform database performance tuning, troubleshooting, and data migration. Providing support to Development and QA teams on applications/database issues Improve system performance by integrating new elements Design and develop scalable and resilient systems using Java or Python to contribute to continual, iterative improvements for product teams Executes ...

Responsibilities: Translate requirements and implement product features using open source technology stack viz. Python with PySpark Research technical feasibility of new functionalities and products Work on Proof of Concept for architectural and design aspect of new functionality Work on continuous improvement of the products through innovation and learning with focus on benchmarking and optimization Design and develop robust databases for real-time, and collaboration platform using Design and develop framework of hierarchy and access control for various resources Required Skills: Senior Python developer to work with existing team building strategic and tactical projects Hands on software en...

Greetings from TCS! TATA Consultancy Services is looking for Big Data Developer with Hands on experience with GCP Required experience range: 6 to 8 years Job locations: Chennai, Gurgaon Job Description: Demonstrated leadership in designing sustainable software products, setting development standards, automated code review process, continuous build and rigorous testing etc- Ability to effectively lead and communicate across 3rd parties, technical and business product managers on solution design- Primary focus is spent writing code, API specs, conducting code reviews & testing in ongoing sprints or doing proof of concepts/automation tools- Applies visualization and other techniques to fast-tra...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Associate Job Description & Summary At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data tra...

Functional Skills Determining, creating, and implementing internal process improvements, such as redesigning infrastructure for increased scalability, improving data delivery, and automating manual procedures. Building analytical tools that make use of the data flow and offer a practical understanding of crucial company performance indicators like operational effectiveness and customer acquisition. Helping stakeholders, including the data, design, product, and executive teams, with technical data difficulties. Working on data-related technical challenges while collaborating with stakeholders, including the Executive, Product, Data, and Design teams, to support their data infrastructure needs...

Roles & Responsibilities Key Responsibilities: Claims Data Analytics & Modeling Analyze medical, pharmacy, and hospital claims data (e.g., CPT, HCPCS, ICD-10, DRG, NPI, etc.). Identify fraud, waste, and abuse (FWA) patterns in claims data. Develop predictive models for risk stratification, readmission prediction, and cost forecasting. Work on value-based care models, risk adjustment, and population health analytics. Perform longitudinal analysis of patient journeys across different providers and payers. Data Engineering & Management Work with large-scale healthcare databases Work with SQL, Python, R, PySpark, and cloud-based big data solutions (AWS, Azure, GCP). Required Skills & Qualificati...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Associate Job Description & Summary At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In business intelligence at PwC, you will focus on leveraging data and analytics to provide strategic insights and drive informed decision-making for clients. You will develop and implement innovative solutions to optimise business performance and enhance competitive advantage. *Why PWC At PwC, you will be part of a...

About the Role We are looking for an experienced Lead Data Engineer to design, build, and optimize cloud-native data platforms on AWS. You will lead a team of Data Engineers and drive end-to-end development of scalable data pipelines, modern data architectures, and enterprise-grade data solutions for BFSI customers. This is a hands-on role with strong ownership across architecture, development, optimisation, and team leadership. Key Responsibilities Leadership & Delivery Lead, mentor, and manage a team of Data Engineers to deliver high-quality outcomes. Own project planning, task delegation, sprint delivery, and quality assurance. Collaborate with cross-functional teams including data scient...