23077 Pyspark Jobs - Page 11

Experience: 6+ Years Domain: Data Engineering / Big Data About the Role: We are seeking a Senior Data Engineer to drive the development of our next-generation Data Lakehouse architecture. You will be responsible for designing, building, and optimizing massive-scale, low-latency data pipelines that support real-time analytics and Machine Learning applications. Key Responsibilities: - Design and build highly optimized, production-grade ETL/ELT pipelines using Apache Spark (PySpark/Scala) to process petabytes of data. - Architect and manage the Data Lakehouse using open-source technologies like Delta Lake or Apache Hudi for ACID transactions and data quality. - Integrate and process real-time d...

Data Engineering concepts (Data Warehouse (DW), Data Lake, OLTP vs OLAP, etc.); API Development (Python); Docker & Kubernetes; Airflow; AWS project;Strong communication skills

Duties & Responsibilities Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis Translate business requirements into well-designed, documented, and reusable data solutions Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi,Databricks etc.) Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability Document pipelines, workflows, and design decisions for knowledge s...

We are seeking a highly skilled and motivated Big Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience with Hadoop ecosystem, Apache Spark, and programming expertise in Python (PySpark), Scala, and Java. You will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions to support analytics and business intelligence initiatives.

Experience: 6 to 10 years of hands-on experience in the data analytics space as a data engineer. Familiar with ETL, DQ, DM, and reject and recycling concepts. A significant portion of this experience should involve building data analytics solutions in a big data environment using Hadoop clusters or cloud environment. Technical Skills: Extensive experience in building data pipelines using Spark, particularly PySpark. Candidates must have a minimum of 3 years of hands-on experience in coding with PySpark applications using RRDs, DataFrames & datasets and NOT Spark SQLs. Candide should has developed numerous spark application for various use cases of processing large volumes of data, used perfo...

About Position: We are looking for Lead Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Azure Data Lead Engineer Location: Pune/ Bangalore Experience: 8 to 10 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tool...

About Position: We are looking for Sr Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team. Role: Sr. Azure Data Engineer Location: Pune/Bangalore Experience: 5 to 8 years Job Type: Full Time Employment What You'll Do: Design and implement complex, scalable data pipelines for ingestion, processing, and transformation using Azure Databricks technologies. Collaborate with Architects, Data Analysts, and Business Analysts to understand data requirements and develop efficient workflows. Develop and manage data storage solutions including Azure SQL Database, Data Lake, and Blob Storage. Leverage Azure Data Factory and other cloud-native tools to ...

Key Responsibilities: Technical Strategy & Roadmap: Define and execute comprehensive ML/AI strategy aligned with business objectives, including LLM infrastructure, retrieval pipelines, and generative workflows Architecture Oversight: Lead system architecture decisions for scalable ML platforms,ensuring robust model deployment and operational excellence Cross-functional Collaboration: Partner closely with product management, engineering, and business stakeholders to translate requirements into technical solutions Model Development: Actively participate in designing, training, and optimizing machine learning models, maintaining technical currency with 20-30% hands-on contribution Code Reviews ...

About the Role We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques. The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work clos...

Design, implement, and maintain data pipelines for processing large datasets, ensuring data availability, quality, and efficiency for machine learning model training and inference. Collaborate with data scientists to streamline the deployment of machine learning models, ensuring scalability, performance, and reliability in production environments. Develop and optimize ETL (Extract, Transform, Load) processes, ensuring data flow from various sources into structured data storage systems. Ensure effective model monitoring, versioning, and logging to track performance and metrics in a production setting. Collaborate with cross-functional teams to improve data architectures and facilitate the con...

Job Title: Data Engineer Azure Fabric, PySpark & Migration Specialist Position Summary We are seeking a Data Engineer with strong expertise in Azure Fabric, PySpark, and data pipeline orchestration to lead the migration of existing tools and workflows from Azure Synapse and SQL Server stored procedures into a modern data platform built on Microsoft Fabric. A deep understanding of distributed data processing, Apache Airflow, and cloud-native data engineering is essential. This role will be pivotal in transforming our data infrastructure, with hands-on responsibilities in pipeline development, data modeling, and performance optimization using tools such as PySpark, Delta Lake, and OneLake. Key...

We are seeking a highly skilled and motivated Big Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience with Hadoop ecosystem, Apache Spark, and programming expertise in Python (PySpark), Scala, and Java. You will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions to support analytics and business intelligence initiatives.

5+ years of experience in Oracle, PL/SQL Develop PL/SQL Packages, Stored Procedures, Functions, Triggers, SQL statements Strong analytical experience with database in writing complex queries, query optimization, debugging, user defined functions, views, indexes Developing scripts for reporting, data manipulation and automation of daily tasks Perform database performance tuning, troubleshooting, and data migration. Providing support to Development and QA teams on applications/database issues Improve system performance by integrating new elements Design and develop scalable and resilient systems using Java or Python to contribute to continual, iterative improvements for product teams Executes ...

Responsibilities: Translate requirements and implement product features using open source technology stack viz. Python with PySpark Research technical feasibility of new functionalities and products Work on Proof of Concept for architectural and design aspect of new functionality Work on continuous improvement of the products through innovation and learning with focus on benchmarking and optimization Design and develop robust databases for real-time, and collaboration platform using Design and develop framework of hierarchy and access control for various resources Required Skills: Senior Python developer to work with existing team building strategic and tactical projects Hands on software en...

Greetings from TCS! TATA Consultancy Services is looking for Big Data Developer with Hands on experience with GCP Required experience range: 6 to 8 years Job locations: Chennai, Gurgaon Job Description: Demonstrated leadership in designing sustainable software products, setting development standards, automated code review process, continuous build and rigorous testing etc- Ability to effectively lead and communicate across 3rd parties, technical and business product managers on solution design- Primary focus is spent writing code, API specs, conducting code reviews & testing in ongoing sprints or doing proof of concepts/automation tools- Applies visualization and other techniques to fast-tra...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Associate Job Description & Summary At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data tra...

Functional Skills Determining, creating, and implementing internal process improvements, such as redesigning infrastructure for increased scalability, improving data delivery, and automating manual procedures. Building analytical tools that make use of the data flow and offer a practical understanding of crucial company performance indicators like operational effectiveness and customer acquisition. Helping stakeholders, including the data, design, product, and executive teams, with technical data difficulties. Working on data-related technical challenges while collaborating with stakeholders, including the Executive, Product, Data, and Design teams, to support their data infrastructure needs...

Roles & Responsibilities Key Responsibilities: Claims Data Analytics & Modeling Analyze medical, pharmacy, and hospital claims data (e.g., CPT, HCPCS, ICD-10, DRG, NPI, etc.). Identify fraud, waste, and abuse (FWA) patterns in claims data. Develop predictive models for risk stratification, readmission prediction, and cost forecasting. Work on value-based care models, risk adjustment, and population health analytics. Perform longitudinal analysis of patient journeys across different providers and payers. Data Engineering & Management Work with large-scale healthcare databases Work with SQL, Python, R, PySpark, and cloud-based big data solutions (AWS, Azure, GCP). Required Skills & Qualificati...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Associate Job Description & Summary At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In business intelligence at PwC, you will focus on leveraging data and analytics to provide strategic insights and drive informed decision-making for clients. You will develop and implement innovative solutions to optimise business performance and enhance competitive advantage. *Why PWC At PwC, you will be part of a...

About the Role We are looking for an experienced Lead Data Engineer to design, build, and optimize cloud-native data platforms on AWS. You will lead a team of Data Engineers and drive end-to-end development of scalable data pipelines, modern data architectures, and enterprise-grade data solutions for BFSI customers. This is a hands-on role with strong ownership across architecture, development, optimisation, and team leadership. Key Responsibilities Leadership & Delivery Lead, mentor, and manage a team of Data Engineers to deliver high-quality outcomes. Own project planning, task delegation, sprint delivery, and quality assurance. Collaborate with cross-functional teams including data scient...

Duties & Responsibilities Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis Translate business requirements into well-designed, documented, and reusable data solutions Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi,Databricks etc.) Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability Document pipelines, workflows, and design decisions for knowledge s...

Airtel is a leading telecom provider with over 400 million subscribers . As a data-driven organization , we process 2 trillion events daily and store 100 PB of data , enabling smarter business decisions. With growing data demands and evolving use cases, we need faster, scalable decision-making powered by analytics and AI. About the role: As a Sr . Lead Data Analyst at Airtel, you will play a pivotal role in driving data-backed decision-making by providing actionable insights across various business functions. You will work independently to analyze complex business data related to sales, revenue, customer experience, and financial operations (billing, payments, collections). You will also con...

Job Title Overview We are looking for an experienced Data Engineer with strong expertise in Databricks and Apache Spark to build and optimize scalable data pipelines on cloud platforms. The ideal candidate will have hands-on experience delivering ETL/ELT workflows, transforming large datasets, and supporting analytics and data platform initiatives. Experience across Azure is required, while exposure to GCP and modern orchestration tools is a plus. Mandatory Skills Strong hands-on experience with Azure Databricks Expert-level proficiency in Apache Spark (PySpark/Scala) Solid understanding of ETL/ELT pipelines, batch & streaming data processing Proficient in Python and SQL Good-to-Have Skills ...

As a Data Intern, you will play a crucial role in supporting our data team with various tasks related to data processing, analysis, and reporting. This position is ideal for a recent graduate or entry-level professional with a strong foundation in Python, PySpark, and SQL, and a keen interest in developing practical skills in a real-world data environment. Key Responsibilities: 1. Data Processing and Analysis: · Contribute to the development and implementation of data processing pipelines using PySpark. · Assist in cleaning, transforming, and analyzing large datasets to extract meaningful insights. 2. Python Programming: · Collaborate with the data team to write Python code for data manipula...

Dear Candidate, Greetings from TCS !!! TCS is hiring for Azure Data Engineer, please find the below JD.. Experience range – 6 to 8 years Location - Pune, Kolkata, Mumbai, Bangalore Skills Required - Azure Databricks, Azure data factory, Azure synapse, Pyspark Required Competencies: 1) Strong design and data solutioning skills 2) Pyspark hands-on experience with complex transformations and large dataset handling experience 3) Good command and hands-on experience in Python. Experience working with following concepts, packages, and tools, a. Object oriented and functional programming b. NumPy, Pandas, Matplotlib, requests, pytest c. Jupyter, PyCharm and IDLE d. Conda and Virtual Environment 4) ...