15947 Pyspark Jobs - Page 42

dunnhumby is the global leader in Customer Data Science, empowering businesses everywhere to compete and thrive in the modern data-driven economy. We always put the Customer First. Our mission: to enable businesses to grow and reimagine themselves by becoming advocates and champions for their Customers. With deep heritage and expertise in retail – one of the world’s most competitive markets, with a deluge of multi-dimensional data – dunnhumby today enables businesses all over the world, across industries, to be Customer First. dunnhumby employs nearly 2,500 experts in offices throughout Europe, Asia, Africa, and the Americas working for transformative, iconic brands such as Tesco, Coca-Cola,...

Job Summary Synechron is seeking a highly skilled and experienced Senior Data Engineer to join our innovative analytics team in Bangalore. The primary purpose of this role is to design, develop, and maintain scalable data pipelines and architectures that empower data-driven decision making and advanced analytics initiatives. As a critical contributor within our data ecosystem, you will enable the organization to harness large, complex datasets efficiently, supporting strategic business objectives and ensuring high standards of data quality, security, and performance. Your expertise will directly contribute to building robust, efficient, and secure data solutions that drive business value acr...

Role & responsibilities We are seeking a highly skilled AWS Data Engineer with 68 years of experience in designing, building, and optimizing scalable data pipelines and cloud-based data platforms. The ideal candidate will have deep expertise in AWS native services, data modeling, ETL/ELT processes, and hands-on experience in enabling analytics solutions. You will collaborate with cross-functional teams including data architects, analysts, and business stakeholders to deliver high-performance, secure, and reliable data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using AWS services (Glue, EMR, Lambda, Step Functions, etc.). Build ...

Role : Azure Data Engineer Type : 6 month contact Location : Chennai, Tamil Nadu (WFO) Experience : 6-10 Years Skills : Pyspark, Databricks, SQL ***Prefer Immediate Joiners*** Job Responsibilities: Design and build ETL/ELT pipelines to ingest, transform, and load data from clinical, omics, research, and operational sources. Optimize performance and scalability of data flows using tools like Apache Spark, Databricks, or AWS Glue. Collaborate with domain experts in genomics, clinical trials, and lab science to understand and implement robust data solutions. Develop and maintain data models, schemas, and governance practices for structured and unstructured biomedical data. Implement data qualit...

Job Title: Cloud Data Engineer Location: Noida, India Experience: 3–6 Years About The Role We are seeking a talented Data Engineer to design, develop, and optimize enterprise-scale data platforms. The role requires strong expertise in Databricks, PySpark, ETL pipelines, GCP services, Unity Catalog, and Terraform . The ideal candidate will collaborate with architects and cross-functional teams to deliver secure, scalable, and high-performance data solutions. Key Responsibilities Design, build, and maintain ETL/ELT pipelines using Databricks and PySpark. Implement and manage data pipelines on Google Cloud Platform (BigQuery, Dataflow, Pub/Sub, etc.). Support data cataloging and governance init...

Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : Databricks Unified Data Analytics Platform, MySQL Good to have skills : PySpark Minimum 2 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As an Application Developer, you will design, build, and configure applications to meet business process and application requirements. A typical day involves collaborating with team members to understand project needs, developing application features, and ensuring that the applications function seamlessly within the business envi...

-Hands-on exp with Hadoop Eco-system (HDFS, Hive, Spark) -Exp on Hive / HDFS, PySpark, Spark, JupiterNotebook, ELT Talend, Control-M, Unix/Script, Python, CI/CD, Git / Jira, Hadoop, TOM, Oozie, Snowflake -Implementation of the Data Quality Controls

7+ Years of expertise required in below skillsets: Data Engineering, Python, SQL, PySpark, Cloud (Any), Airflow, DevOps & Orchestration, Agile, Scrum, Data Pipeline Development, Stakeholder Management, Banking/ Telecom Domain Experience.

Role & responsibilities Design and implement scalable data pipelines for batch and near real-time use cases in cloud environments Write optimized, complex SQL queries for data transformation and analysis in cloud data warehouses Develop efficient Python and PySpark scripts for large-scale data processing and ETL workflows Create and maintain data models in Databricks , ensuring data quality and consistency Optimize queries and scripts over large-scale datasets (TBs) with a focus on performance and cost-efficiency Implement data governance and security best practices in cloud environments Collaborate across teams to translate business requirements into robust technical solutions Preferred can...

LLM, Generative AI, Python, LLM frameworks, LangChain, Hugging Face, OpenAI SDK, vector databases, FAISS, Pinecone, Weaviate, AWS, Databricks, PySpark, SQL, Git, CI/CD pipelines, RAG architecture, prompt engineering, token optimization,AI frameworks

Job Location : Remote Job Timings : India Evening Shift (till 11 : 30 PM IST) We are seeking a skilled Databricks Developer to join our team on a 4-month contract basis. The ideal candidate will have strong expertise in modern data engineering technologies and the ability to work in a fast-paced, remote environment. Key Responsibilities : - Develop, optimize, and manage large-scale data pipelines using Databricks, PySpark, DBT, and AWS S3/Glue/Redshift. - Work in multi-cloud environments including AWS, Azure, and GCP. - Implement workflow orchestration using Airflow or similar frameworks. - Design, implement, and manage data warehouse solutions, schema evolution, and data versioning. - Colla...

Exp: 6 to 10 years Job Location: Noida / Mumbai / Pune / Bangalore / Gurugram / Kochi (Hybrid WFO) Shift Timing : 7 am to 4 pm Notice : Immediate to 30 days Role & responsibilities In this role, you will support the Data Engineering team in setting up the Data Lake on Cloud and the implementation of standardized Data Model, single view of customer. You will develop data pipelines for new sources, data transformations within the Data Lake , implementing graphql , work on no sql database, CI/CD and data delivery as per the business requirements. Job Description: • Build pipelines to bring in wide variety of data from multiple sources within the organization as well as from social media and pub...

We are looking for a skilled Machine Learning Engineer with expertise in Azure and Databricks. The candidate will bring experience in building, training, and deploying ML models in real-world applications.

Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : Apache Spark Good to have skills : Python (Programming Language), Google BigQuery, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As an Application Developer, you will design, build, and configure applications to meet business process and application requirements. A typical day involves collaborating with various teams to gather requirements, developing application features, and ensuring that the applications are aligned with business objectives....

🚀 We’re Hiring: Data Engineer | Xebia 📍 Location : Any Xebia location (Hybrid, 3 days office per week) 🕒 Experience : 6+ years ⏳ Notice Period : Immediate to 2 weeks – only apply if you can join early We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, SQL, and Postgres . The ideal candidate will also have hands-on experience with Vector Databases (pgvector, Qdrant, Pinecone, etc.) and exposure to Generative AI use cases such as RAG pipelines and embedding-based search . ✅ Key Skills & Expertise Proficiency in Python for data engineering (PySpark, pandas, APIs). Strong SQL skills for data modeling, optimization, and analytics. Hands-on expertise in Databr...

Why Join Iris? Are you ready to do the best work of your career at one of India’s Top 25 Best Workplaces in IT industry ? Do you want to grow in an award-winning culture that truly values your talent and ambitions ? Join Iris Software — one of the fastest-growing IT services companies — where you own and shape your success story . About Us At Iris Software, our vision is to be our client’s most trusted technology partner, and the first choice for the industry’s top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transpo...

Role Description REQUIREMENT: Professional experience of 3-5 years in business analysis, requirement gathering, solution workflow design for AI/ML/Analytics project Experience in programming skills in Python and SQL Strong understanding of ML libraries and applications e.g., Time series analysis, Neural Net, SVMs, boosting methods and implementation using Python Experience in Deep learning techniques Excellent Communication skills to effectively collaborate with business SMEs, UX team Intermediate knowledge with Dev Ops practices, including CICD, Automation, Build Management, and Versioning Understanding of Cloud services, including Microsoft Azure, GCP or Amazon Web Service Experience in LL...

Role Description Data Scientist REQUIREMENT: Professional experience of 3-5 years in business analysis, requirement gathering, solution workflow design for AI/ML/Analytics project Experience in programming skills in Python and SQL Strong understanding of ML libraries and applications e.g., Time series analysis, Neural Net, SVMs, boosting methods and implementation using Python Experience in Deep learning techniques Excellent Communication skills to effectively collaborate with business SMEs, UX team Intermediate knowledge with Dev Ops practices, including CICD, Automation, Build Management, and Versioning Understanding of Cloud services, including Microsoft Azure, GCP or Amazon Web Service E...

Role Description Data Scientist REQUIREMENT: Professional experience of 3-5 years in business analysis, requirement gathering, solution workflow design for AI/ML/Analytics project Experience in programming skills in Python and SQL Strong understanding of ML libraries and applications e.g., Time series analysis, Neural Net, SVMs, boosting methods and implementation using Python Experience in Deep learning techniques Excellent Communication skills to effectively collaborate with business SMEs, UX team Intermediate knowledge with Dev Ops practices, including CICD, Automation, Build Management, and Versioning Understanding of Cloud services, including Microsoft Azure, GCP or Amazon Web Service E...

Requirement Role description Professional experience of 15+ years in business analysis, requirement gathering, solution workflow design for AI/ML/Analytics project Experience in programming skills in Python and SQL Strong understanding of ML libraries and applications e.g., Time series analysis, Neural Net, SVMs, boosting methods and implementation using Python Experience in Deep learning techniques Excellent Communication skills to effectively collaborate with business SMEs, UX team Intermediate knowledge with Dev Ops practices, including CICD, Automation, Build Management, and Versioning Understanding of Cloud services, including Microsoft Azure, GCP or Amazon Web Service Experience in LLM...

🌟 We're Hiring: PySpark Data Engineer! 🌟 We are seeking an experienced PySpark Data Engineer to design, develop, and maintain large-scale data processing systems. The ideal candidate will have expertise in Apache Spark, Python, and big data technologies to build robust data pipelines and analytics solutions. 📍 Location: Hyderabad, India ⏰ Work Mode: Flexible office & remote 💼 Role: PySpark Data Engineer What You'll Do 🎯 Design and develop scalable data pipelines using PySpark 📊 Optimize data processing workflows for performance and reliability 🔧 Build ETL processes for large datasets across multiple sources ☁️ Deploy and maintain data solutions on cloud platforms 📈 Collaborate with data scie...

Role Description UST is looking for a talented GCP Data Engineer with 5 to 10 years of experience to join our team and play a crucial role in designing and implementing efficient data solutions on the Google Cloud Platform (GCP). The ideal candidate should possess strong data engineering skills, expertise in GCP services, and proficiency in data processing technologies, particularly PySpark. Responsibilities Data Pipeline Development: Design, implement, and optimize end-to-end data pipelines on GCP, focusing on scalability and performance. Develop and maintain ETL workflows for seamless data processing. GCP Cloud Expertise Utilize GCP services such as BigQuery, Cloud Storage, and Dataflow fo...

Role Description Job Title: Data Engineer Experience : 5-8 years Job Description We are seeking a highly skilled Data Engineer with expertise in building scalable data pipelines and working with Azure Cloud services. The ideal candidate will be proficient in designing and implementing end-to-end data pipelines, ensuring high performance and scalability. You will leverage Azure services like Azure Data Factory, Azure SQL Database, and Azure Databricks, as well as PySpark for data transformations. This is an excellent opportunity to make a significant impact in an innovative environment. Key Responsibilities Data Pipeline Development: Design, implement, and optimize end-to-end data pipelines o...

Role Description Key Responsibilities Design, develop, and optimize ETL pipelines using PySpark on Google Cloud Platform (GCP). Work with BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration. Develop and optimize Spark-based ETL processes for large-scale data processing. Implement best practices for data governance, security, and monitoring in a cloud environment. Collaborate with data engineers, analysts, and business stakeholders to understand data requirements. Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution. Automate data workflows using Apache Airflow or Cloud Composer. Ensure data qua...

Role Description Key Responsibilities: Big Data Architecture: Design, develop, and maintain scalable and distributed data architectures capable of processing large volumes of data. Data Storage Solutions: Implement and optimize data storage solutions using technologies such as Hadoop, Spark, and PySpark. PySpark Development: Develop and implement efficient ETL processes using PySpark to extract, transform, and load large datasets. Performance Optimization: Optimize PySpark applications for better performance, scalability, and resource management. Qualifications Proven experience as a Big Data Engineer with a strong focus on PySpark. Deep understanding of Big Data processing frameworks and te...