1569 Data Pipeline Jobs - Page 11

We are looking for a skilled Data Engineer with expertise in PySpark, AWS, and SQL to support data processing and analytical initiatives. This role involves working closely with data engineering and data science teams to build, maintain, and optimize large-scale data pipelines and integrations on AWS. The ideal candidate will be proficient in ETL processes using PySpark and SQL, with a deep understanding of cloud data infrastructure, specifically within the AWS ecosystem. Key Responsibilities : Data Pipeline Development : - Design, build, and optimize ETL pipelines using PySpark for data ingestion, transformation, and storage on AWS. - Collaborate with stakeholders to understand data require...

Skill: AWS Glue Experience: 9 to 12 years Location: AIA Chennai , AIA Bangalore Job Summary Join our dynamic team as an Architect where you will leverage your expertise in Spark Optimization AWS DevOps and Python to design and implement cutting-edge solutions. With a focus on AWS Glue Studio AWS Glue ETL and Amazon S3 you will drive innovation and efficiency in our hybrid work model. This role requires a minimum of 7 years of experience offering an exciting opportunity to make a significant impact. Responsibilities Design and implement scalable data architectures using Spark Optimization to enhance performance and efficiency. Collaborate with cross-functional teams to integrate AWS DevOps pr...

AIA - Face to Face Drive - Chennai Skill: GCP, BigQuery & Python Experience: 4 to 9 Years Location: Chennai Drive Date: 06 Dec 2025 Job Summary Data Engineer with a focus on Google Cloud Platform (GCP). Hands-on experience with GCP services: Dataflow Pub/Sub Cloud Storage Cloud Composer. Expertise in workflow orchestration using Apache Airflow. Experience with cloud SQL or other relational databases (PostgreSQL preferred). Solid understanding of data modeling ETL processes and streaming/batch data pipelines. Familiarity with DevOps practices CI/CD and containerization (Docker Kubernetes). Responsibilities Key Responsibilities and Skills Build Data Pipelines on GCP: They need hands-on experie...

Retail Media is transforming how advertisers connect with consumers through personalized and targeted campaigns across retailers' digital and physical touchpoints. Retail Media Measurement plays a pivotal role in ensuring the effectiveness of these campaigns, driving value for advertisers, retailers, and consumers alike. This role focuses on designing, building, and scaling solutions that enable the accurate measurement of retail media campaigns across various channels. By providing actionable insights, it empowers stakeholders to optimize media investments, improve ROI, and enhance the overall customer experience. Job Title: Senior Data Engineer Job Summary We are looking for a talented and...

Role: Data Engineer Location: Noida JD: We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way. Key Responsibilities: Data Pipeline Development: Design, build, and maintain robust, scalable, and effic...

5+ years of QA experience with a focus on automation and cloud-native environments. Hands-on expertise in automation tools such as Selenium, Cypress, or Playwright. Experience with ETL/data pipeline testing, including validating source-to-target data mapping, transformations, and aggregation logic. Proficient in writing and executing complex SQL queries for data validation. Working knowledge of GCP services such as BigQuery, Pub/Sub, Cloud Functions, Cloud Storage, etc. Familiarity with API testing using Postman, RestAssured, or similar tools. Scripting knowledge in Python, JavaScript, or Java for test automation. Experience working in CI/CD pipelines and version control (Git). Roles and Res...

We are looking for an experienced GCP (Google Cloud Platform), BigQuery, and Python Developer to join our team. The ideal candidate will have extensive experience with Google Cloud technologies, specifically BigQuery, and a strong background in Python programming. This role requires hands-on experience in designing, developing, and maintaining data solutions on GCP, with a focus on data warehousing, ETL processes, and data analytics. Key Responsibilities : - Design and implement data pipelines using GCP services, particularly BigQuery, for efficient data processing, analysis, and storage.- Develop Python-based scripts and applications to automate data extraction, transformation, and loading ...

We are currently seeking a skilled Data Engineer with expertise in Talend to join our team. The ideal candidate will have 4 years of overall experience in data engineering and possess a strong proficiency in Talend tools and technologies. This role will involve working with large datasets, designing data integration solutions, and ensuring the smooth flow of data across various systems. Responsibilities : - Develop and implement data integration solutions using Talend tools. - Design, build, and maintain scalable data pipelines for data processing, transformation, and storage. - Collaborate with cross-functional teams to gather data requirements and understand business needs. - Perform data ...

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure. Key Responsibilities : SQL Development : - Write medium-complexity SQL queries to extract, transform, and load data efficiently. - Optimize SQL queries for performance and scalability. - Collaborate with team members to understand data requirements and translate them...

Contract Duration : 3 months Skills : - PySpark or Scala with Spark, Spark Architecture, Hadoop, SQL - Streaming Technologies like Kafka etc.- Proficiency in Advanced SQL (Window functions)- Airflow, S3, and Stream Sets or similar ETL tools.- Basic Knowledge on AWS IAM, AWS EMR and Snowflake. Responsibilities : - Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of structured and unstructured data. - Data Modeling: Develop and maintain data models, schemas, and metadata to support the organization's data initiatives. Ensure data integrity and optimize data storage and retrieval processes. - Data Integ...

About Cloudaeon: Cloudaeon is a global technology consulting and services company helping enterprises modernize and manage their cloud infrastructure, big data ecosystems, DevOps pipelines, and advanced analytics platforms . We deliver high-quality, scalable solutions built on deep domain expertise and leading enterprise data technologies. Our teams work actively on AI and GenAI initiatives , incorporating machine learning, intelligent automation, and modern data engineering practices to create future-ready digital solutions. With a highly skilled global team committed to excellence, we consistently exceed customer expectations and help organizations achieve their strategic goals. Job Role A...

Responsibilities: Design, develop & maintain scalable backends using Python, ETL, Snowflake & Microservices. Collaborate with teams on data integration & pipeline development. Diagnose,resolve technical issues maintain Python application reliability. Health insurance Performance bonus

Job Summary Join our dynamic team as a Sr AI Engineer where you will leverage cutting-edge technologies to drive innovation and deliver impactful solutions. With a focus on AWS Machine Learning GenAI and other advanced tools you will play a crucial role in transforming data into actionable insights. This hybrid role offers the opportunity to work with a talented team in a day shift environment without the need for travel. Responsibilities Develop and implement machine learning models using AWS Machine Learning and Amazon Sagemaker to solve complex business problems and enhance decision-making processes. Utilize Neuro AI and Open AI Service to create innovative solutions that improve customer...

Design, develop & maintain relational (SQL) & non-relational (NoSQL) databases. Strong proficiency in PostgreSQL / MySQL / Oracle / SQL Server & MongoDB/ Document DB. Hands on experience on query optimization, indexing strategies & performance tuning. Plan, design & execute end-to-end data migration strategies from legacy to modern databases. Good knowledge of data modelling, data warehouse concept, database design, data pipeline. Develop ETL pipelines using SQL, Python or tools like Airflow, Talend etc. Develop python scripts for automation, ingestion & integration with database (SQL + NoSQL) Willing to work Individually on development/support projects. Critical thinker and problem-solving ...

Senior Data Engineering Manager / Director will own end-to-end data onboarding function within SaaS platform for self-storage industry. scaling the data eng & migration frameworks that enable properties PMS systems into Hummingbird. Required Candidate profile data migration pipeline from PMS Hummingbird onboarding teams, Product, Engineering and Sales teams to design repeatable onboarding playbooks for PMS migrations.

Societe Generale Global Solution Centre is looking for Lead Data Engineer - Spark, Scala to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Lead Data Engineer with expertise in Spark and Scala to join our team at Societe Generale Global Solution Centre. The ideal candidate will have a strong background in data engineering and experience working with large datasets. Roles and Responsibility Design, develop, and implement scalable data pipelines using Spark and Scala. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems that meet business needs. Troubleshoot and resolve...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – Python, AWS to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and AWS to join our team. The ideal candidate will have a strong background in software development and experience working with cloud-based technologies. Roles and Responsibility Design, develop, and test software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines using AWS services such as S3, Lambda, and EMR. Ensure high-quality code by following best practices and test...

EPAM Anywhere is looking for Senior Software Engineer – (Python with LLM/GenAI) to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and LLM/GenAI to join our team. The ideal candidate will have a strong background in software development and experience working with large language models. Roles and Responsibility Design, develop, and deploy scalable software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines and architectures. Work on natural language processing tasks such as text classific...

EPAM Anywhere is looking for Senior Software Engineer – (Python with LLM/GenAI) to join our dynamic team and embark on a rewarding career journey We are looking for a skilled Senior Software Engineer with expertise in Python and LLM/GenAI to join our team. The ideal candidate will have a strong background in software development and experience working with large language models. Roles and Responsibility Design, develop, and deploy scalable software applications using Python. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data pipelines and architectures. Work on natural language processing tasks such as text classific...

Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be skilled in ETL tools such as Informatica Glue Databricks and DataProc with coding expertise in Python PySpark and SQL. Works independently and has a deep understanding of data warehousing solutions including Snowflake BigQuery Lakehouse and Delta Lake. Capable of calculating costs and understanding performance issues related to data solutions. Outcomes: Act creatively to develop pipelines and applications by selecting appropriate technical options optimizing application development mainten...