22907 Pyspark Jobs - Page 5

Duties & Responsibilities Design, develop, and maintain scalable ETL/ELT data pipelines to support business and analytics needs Write, tune, and optimize complex SQL queries for data transformation, aggregation, and analysis Translate business requirements into well-designed, documented, and reusable data solutions Partner with analysts, data scientists, and stakeholders to deliver accurate, timely, and trusted datasets Automate data workflows using orchestration/scheduling tools (Airflow, ADF, Luigi,Databricks etc.) Develop unit tests, integration tests, and validation checks to ensure data accuracy and pipeline reliability Document pipelines, workflows, and design decisions for knowledge s...

Airtel is a leading telecom provider with over 400 million subscribers . As a data-driven organization , we process 2 trillion events daily and store 100 PB of data , enabling smarter business decisions. With growing data demands and evolving use cases, we need faster, scalable decision-making powered by analytics and AI. About the role: As a Sr . Lead Data Analyst at Airtel, you will play a pivotal role in driving data-backed decision-making by providing actionable insights across various business functions. You will work independently to analyze complex business data related to sales, revenue, customer experience, and financial operations (billing, payments, collections). You will also con...

Job Title Overview We are looking for an experienced Data Engineer with strong expertise in Databricks and Apache Spark to build and optimize scalable data pipelines on cloud platforms. The ideal candidate will have hands-on experience delivering ETL/ELT workflows, transforming large datasets, and supporting analytics and data platform initiatives. Experience across Azure is required, while exposure to GCP and modern orchestration tools is a plus. Mandatory Skills Strong hands-on experience with Azure Databricks Expert-level proficiency in Apache Spark (PySpark/Scala) Solid understanding of ETL/ELT pipelines, batch & streaming data processing Proficient in Python and SQL Good-to-Have Skills ...

As a Data Intern, you will play a crucial role in supporting our data team with various tasks related to data processing, analysis, and reporting. This position is ideal for a recent graduate or entry-level professional with a strong foundation in Python, PySpark, and SQL, and a keen interest in developing practical skills in a real-world data environment. Key Responsibilities: 1. Data Processing and Analysis: · Contribute to the development and implementation of data processing pipelines using PySpark. · Assist in cleaning, transforming, and analyzing large datasets to extract meaningful insights. 2. Python Programming: · Collaborate with the data team to write Python code for data manipula...

Dear Candidate, Greetings from TCS !!! TCS is hiring for Azure Data Engineer, please find the below JD.. Experience range – 6 to 8 years Location - Pune, Kolkata, Mumbai, Bangalore Skills Required - Azure Databricks, Azure data factory, Azure synapse, Pyspark Required Competencies: 1) Strong design and data solutioning skills 2) Pyspark hands-on experience with complex transformations and large dataset handling experience 3) Good command and hands-on experience in Python. Experience working with following concepts, packages, and tools, a. Object oriented and functional programming b. NumPy, Pandas, Matplotlib, requests, pytest c. Jupyter, PyCharm and IDLE d. Conda and Virtual Environment 4) ...

Job Title: Data Engineer (AWS + Databricks + PySpark) Experience: 5–12 years Location: Pan India Notice Period: Immediate joiners only Job Description: Looking for a Data Engineer with strong experience in AWS, Databricks, PySpark, and SQL. The role involves building and maintaining data pipelines, ETL processes, and cloud-based data solutions. Required Skills: AWS (S3, Glue, Lambda, EMR, EC2) Databricks PySpark SQL (must-have) ETL/ELT development Data pipeline design and optimization Data warehousing concepts Good to Have: Python scripting CI/CD tools (Git, Jenkins) Delta Lake or Lakehouse knowledge

About the Company Our client is a trusted global innovator of IT and business services, present in 50+ countries. They specialize in digital & IT modernization, consulting, managed services, and industry-specific solutions. With a commitment to long-term success, they empower clients and society to move confidently into the digital future. Position: Data Engineer Experience: 6+ Years Location: Hyderabad Employment Type: Full-time About the Role We are looking for a highly skilled Data Engineer with 6+ years of experience in designing and building scalable data pipelines, big data processing solutions, and cloud-based data platforms. The ideal candidate will have strong expertise in Big Data ...

TCS present an excellent opportunity for Data Engineer Job Location: Kochi (Only) Experience required :7- 12 yrs Skills: Python, Pyspark, SQL, GCP Walk in Interview Date: 22-Nov-25 (Saturday) Address: TCS Center, Infopark Kochi Phase 1, Kochi – 682042 Must-Have Strong development experience in Python, PySpark, SQL Experience with cloud platforms (GCP preferred /Azure) Experience working in Agile delivery environment. Experience with orchestration tools like Airflow, ADF. Experience with GCP services (GCS Bucket, Cloud Functions, Dataproc, Dataflows, Pub-Sub, Big Query). OR Experience with Azure services (ADLS, Azure Functions, Databricks, Azure Event Hubs, Kafka, Spark Streaming, Azure Synap...

Azure Databricks Architect - Noida, Pune,Chennai,Bangalore,Hyderabad Years of experience - 9-16 years Notice period- Immediate to Serving to 30 days Role Overview: We are looking for experienced Data Architects / Senior Data Architects to join our team. In this role, you will lead the architecture, design, and delivery of modern data platforms—including Data Lakes, Lakehouses, and Data Mesh—using Azure and Databricks. This is a hybrid role involving hands-on development, customer engagement, and technical leadership, where you will collaborate across teams to drive scalable and innovative data solutions end-to-end. Key Responsibilities: Architect and implement data solutions leveraging Azure...

Location: Bangalore - Karnataka, India - EOIZ Industrial Area Job Family: Artificial Intelligence & Machine Learning Worker Type Reference: Regular - Permanent Pay Rate Type: Salary Career Level: T3(B) Job ID: R-49759-2025 Description & Requirements Essential Skills & Experience:Experience Level: 8+ years’ experience inData Engineering development with end-to-end Data Platform implementation using AWS tech stack. Mandatory Skills:- Excellent understanding and design experience of Big Data platform, Data Lake, Lakehouse architecture, Medallion architecture and modern data architecture.- Excellent in SQL/PL-SQL scripting in columnar database- Excellent Data Modelling experience- Strong skills ...

Role: Analytics Consultant Experience Required: 8 - 12 Years Location: Bangalore, Hyderabad, Mumbai, Chennai Employment Type: Full-time Notice period- Immediate to Serving to 30 days Work closely with CPG/FMCG clients to understand business challenges and translate them into analytical solutions. Develop, implement, and optimize machine learning models (e.g., regression, classification, forecasting, clustering, recommendation engines). Perform advanced data analysis, including market mix modelling, demand forecasting, price/elasticity analysis, customer segmentation, sales analytics, and supply chain analytics. Build analytical workflows, dashboards, and automated solutions using Python and ...

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Data Engineer in India. In this role, you will design, build, and maintain scalable data pipelines and systems to support data-driven initiatives across the organization. You will collaborate closely with data scientists, analysts, and other stakeholders to ensure data is accurate, accessible, and actionable. Your work will contribute to improving data architecture, performance, and insights, helping the business make informed decisions. You will have the opportunity to work with cloud-native tools, modern ETL frameworks, and advanced analytics platforms, all within a fast-paced, collab...

We’re looking for a Data Engineer (2–3 years) with strong hands-on experience in Azure Data Factory, Azure Storage, SQL and ETL/ELT pipelines. The ideal candidate can independently build & maintain ADF pipelines, manage data flows, and ensure smooth data movement across Azure services, who have worked closely in connecting to multiple data sources & transforming data & storing without any manual intervention What you’ll do: Build/modify ADF pipelines, datasets, linked services Work with Copy Activity, Mapping Data Flows, triggers, Integration Runtime Ingest and transform data using Azure SQL / Synapse Implement incremental loads, data quality checks, and ETL best practices Handle Blob/ADLS s...

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Data Operations Engineer 2 in India. In this role, you will design, build, and maintain robust, scalable data pipelines and infrastructure to support analytics and business operations in a fast-paced healthcare technology environment. You will work closely with data scientists, analysts, and engineering teams to ensure data is accurate, accessible, and optimized for reporting and operational use. This position provides the opportunity to work with modern cloud technologies, ETL frameworks, and streaming platforms, while contributing to the continuous improvement of data architecture and operat...

Note : Please do not apply if your salary expectations are higher than the provided Salary Range and experience less than 3 years. If you have experience with Travel Industry and worked on Hotel, Car Rental or Ferry Booking before then we can negotiate the package. Company Description Our company is involved in promoting Greece for the last 25 years through travel sites visited from all around the world with 10 million visitors per year such www.greeka.com, www.ferriesingreece.com etc Through the websites, we provide a range of travel services for a seamless holiday experience such online car rental reservations, ferry tickets, transfers, tours etc….. Role Description We are seeking a highly...

Job Description: Data Engineer 3–5 Years Experience Location: Remote RedPluto Analytics is seeking an experienced Data Engineer who is passionate about designing, developing, and optimizing scalable data pipelines. The ideal candidate will have strong hands-on expertise in modern data engineering tools, frameworks, and cloud platforms. Key Responsibilities Design, develop, and maintain scalable data pipelines using PySpark and Big Data technologies. Develop, schedule, and manage workflows using Apache Airflow . Work on data ingestion, transformation, and storage using Data Lakes and Data Warehouses. Optimize PySpark jobs for improved performance and cost efficiency. Implement real-time and b...

Primary skills : Gemini Agent space, AI multi cloud experience, Experience in running AI services, vibe coding(Cursor), Vertex Secondary skills : Terraform, Multi cloud experience (AWS) Excellent programming skills and proficiency in at least one of the major programming scripting languages used in Gen AI orchestration such as Python or PySpark or Java Hands on exposure to integrating with Gemini Pro 1.x using API endpoints. Hands on exposure to using GCP services for storage, serverless-logic, search, transcription, and chat Thorough understanding of Cursor coding(Vibe coding); implementation exposure to LLM agents like LangChain & vector databases Pinecone or Chroma or FAISS Ability to qui...

Roles And Responsibilities Proficiency in building highly scalable ETL and streaming-based data pipelines using Google Cloud Platform (GCP) services and products like Biquark, Cloud Dataflow Proficiency in large scale data platforms and data processing systems such as Google Big Query, Amazon Redshift, Azure Data Lake Excellent Python, PySpark and SQL development and debugging skills, exposure to other Big Data frameworks like Hadoop Hive would be added advantage Experience building systems to retrieve and aggregate data from event-driven messaging frameworks (e.g. RabbitMQ and Pub/Sub) Secondary Skills : Cloud Big Table, AI/ML solutions, Compute Engine, Cloud Fusion (ref:hirist.tech)

Role Overview: As a Data Engineer at our company, you will be responsible for leveraging your expertise in PySpark, Python, SQL, Git, and AWS to design, develop, and maintain our data infrastructure. You will play a crucial role in supporting our analytics by processing and engineering data effectively. Key Responsibilities: - Utilize PySpark, Python, SQL, Git, and AWS services such as Glue, Lambda, Step Functions, S3, and Athena to design and develop data infrastructure. - Implement data warehousing concepts and data modeling to enhance data processing capabilities. - Work with big data technologies like Hadoop and Spark to optimize data workflows. - Obtain AWS certifications related to dat...

As a highly skilled and strategic Data Architect with expertise in the Azure Data ecosystem, your role at Lexmark will involve defining and driving the overall Azure-based data architecture strategy aligned with enterprise goals. You will architect and implement scalable data pipelines, data lakes, and data warehouses using Azure Data Lake, ADF, and Azure SQL/Synapse. Providing technical leadership on Azure Databricks for large-scale data processing and advanced analytics use cases is a crucial aspect of your responsibilities. Your key responsibilities will include: - Integrating AI/ML models into data pipelines and supporting the end-to-end ML lifecycle including training, deployment, and m...

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. - Design and build efficient data pipelines using Azure Databricks (PySpark) - Implement business logic for data transformation and enrichment at scale - Manage and optimize Delta Lake storage solutions - Develop REST APIs using FastAPI to expose processed data and depl...

Role Overview: As a Data Engineer at our company, you will be responsible for developing, optimizing, and managing robust ETL pipelines to handle large-scale data processing and transformation. You will be working on building and maintaining data lake architectures on AWS S3 and data warehouse solutions on Redshift for efficient storage and retrieval. Additionally, you will use Databricks for advanced data processing, pipeline orchestration, and integration with downstream AI/ML systems. Your role will also involve leveraging various AWS services for scalable, production-grade data solutions and developing data processing logic in Python, PySpark, SQL, and other programming languages. Key Re...

Relevant experience in Pyspark and Azure Databricks. Proficiency in integrating, transforming, and consolidating data from various structured and unstructured data sources. Good experience in SQL or native SQL query languages. Strong experience in implementing Databricks notebooks using Python. Good experience in Azure Data Factory, ADLS, Storage Services, Serverless architecture, Azure functions. Exposure to basic issues in working within a Cloud (Azure) environment. Advanced working SQL knowledge and experience working with relational/non-Relational databases, query. Exposure to Jira and Confluence required

Role & responsibilities Job Title: Data Engineer Experience: 3 - 7 years Location: Hybrid Employment Type: Full-time About the Role We are looking for a Data Engineer who will play a key role in designing, building, and maintaining robust data pipelines that power our AI-based analytics and dashboarding solutions. The candidate should have a strong foundation in data modeling, ETL workflows, and cloud-based data infrastructure. Youll work closely with analysts, AI engineers, and business stakeholders to ensure data reliability, scalability, and performance across all projects. Key Responsibilities Design, develop, and maintain data pipelines to collect, clean, and process structured and unst...

As a Manager at EY in the Consulting Microsoft Fabric team, your role will involve designing, developing, and maintaining distributed systems using Microsoft Fabric, including One Lake, Azure Data Factory (ADF), Azure Synapse, Notebooks, Data Warehouse, and Lakehouse. You will play a crucial role in architecting and implementing enterprise data platforms and data management practices, ensuring the delivery of high-quality solutions that meet business requirements. Collaborating with system architects, business analysts, and stakeholders is essential to understand their requirements and convert them into technical designs. Your responsibilities will include designing, building, testing, deplo...