3177 Data Pipelines Jobs - Page 3

Some Careers Shine Brighter Than Others. If you're looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further. HSBC is one of the largest banking and financial services organisations in the world, with operations in 64 countries and territories. We aim to be where the growth is, enabling businesses to thrive and economies to prosper, and, ultimately, helping people to fulfil their hopes and realise their ambitions. We are currently seeking an experienced professional to join our team i...

What are we building We are building a full-stack AI Agentic Platform that powers customer, sales and internal workflows end-to-end. Core capabilities are including but not limited to: Multi-agent, multi-modal customer interactions. Agents which can handle agent decision hubs. Evaluation engines to score and diagnose interactions. Generating insights from chat to improve SOP, Tools and Product Flow Fine-tuning & deployment pipelines Observability with agent telemetry systems Safety, governance, and cost optimization layer. What are we looking for We are looking for builders who have a passion for building great products and have demonstrated this through their projects in recent years. Compe...

Role Title: Lead Engineer - SAP Data & Analytics Location: Chennai We are one purpose-led global organisation. The enablers and innovators, ensuring that we can fulfil our mission to push the boundaries of science and discover and develop life-changing medicines. We take pride in working close to the cause, opening the locks to save lives, ultimately making a massive difference to the outside world. AstraZeneca (AZ) is in a period of strong growth and our employees have a united purpose to make a difference to patients around the world who need both our medicines and the ongoing developments from our science. In this journey AZ must continue to work across borders and with partners and new c...

Role Summary We are looking for a Travel Analytics professional with 510 years of experience to own end-to-end travel data analytics, insights, and reporting. This role will lead large-scale travel data processing, build scalable analytics solutions, partner with internal stakeholders, and work closely with Travel Management Companies (TMCs) and related vendors to improve data quality, visibility, and business outcomes. Key Responsibilities Own end-to-end travel analytics: ingest, clean, standardize, validate, and analyze large travel datasets (bookings, ticketing, hotel, car, expense, and traveler profiles). Build and maintain scalable data pipelines and data models to support recurring and...

Overview The next great shift in consumer behavior driven by technological disruption is underway, and its happening in the food & beverage industry. As other sectors have shifted to Digital Commerce-first business models in recent years, food & beverage has continued to rely predominantly on traditional brick & mortar models, but this is changing rapidly. New technologies are transforming every aspect of reaching consumers, from the rise of digital marketing and online grocery platforms to the creation of supply chain tools that enable speedy at-home delivery. Although PepsiCo is a large multinational, the PepsiCo Global Digital Commerce Team prides itself on having the entrepreneurial, act...

Must Have Skills: (80% Skywise) Strong experience building dashboards and KPI reports on Skywise / Palantir Foundry (Workshop, Slate, Quiver, Carbon, Pipelines, Contour) Data analytics experience with mediumlarge datasets Knowledge of Ontology tools (Ontology Manager, Object Explorer, Actions) Strong understanding of data modeling, governance, and data quality Ability to automate reporting and manage end-to-end dashboard development in Skywise Your future duties and responsibilities: Build and maintain dashboards, KPI metrics, and performance reports in Skywise Automate data pipelines and reporting processes Manage data ingestion from multiple sources and ensure data quality & governance Del...

Key Responsibilities Automate assessment data pipelines, validation, and reporting. Build repeatable scripts/workflows using Python, Terraform, or GCP tools. Enhance RaMP tooling for faster assessment cycles. Ensure accuracy, repeatability, and compliance in automation flows. Required Skills & Qualifications 4-7 years in automation engineering or DevOps. Strong GCP SDK, APIs, Terraform, Cloud Build, Cloud Functions. Experience with automation frameworks and scripting.

Design, develop, and maintain data pipelines and ETL processes using AWS and Snowflake Implement data transformation workflows using DBT (Data Build Tool) Write efficient, reusable, and reliable code in Python Optimize and tune data solutions for performance and scalability Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions Ensure data quality and integrity through rigorous testing and validation Stay updated with the latest industry trends and technologies in data engineering Bachelor''s or Master''s degree in Computer Science, Engineering, or a related field Proven experience as a Data Engineer or similar role Strong pro...

We are seeking a highly skilled and experienced Consultant to join our team In this role, you will be responsible for designing and developing robust data pipelines using Python, Apache Spark on Databricks You will also be responsible for optimizing and troubleshooting existing pipelines for performance and reliability As a Consultant, you will collaborate with data scientists and analysts to understand their requirements and ensure data quality, integrity, and security across various sources Additionally, you will implement ETL processes and integrate with cloud platforms such as Azure, AWS, and GCP Other responsibilities include maintaining and monitoring Databricks clusters and notebooks,...

Expert-level Python programming skills with solid understanding of algorithms, data structures, and OOP. Strong understanding of LLMs (Large Language Models) and transformer-based architectures . Hands-on experience with LangChain, LangGraph , and other LLM orchestration frameworks. Proficiency in RAG (Retrieval-Augmented Generation) design and implementation. Experience in document ingestion , data embedding , and vector database management (FAISS, Pinecone, Chroma, Weaviate). Strong grounding in Deep Learning, Machine Learning, and Generative AI . Familiarity with TensorFlow, PyTorch, and Hugging Face ecosystems . Experience in MLOps, Docker, Kubernetes, and CI/CD pipelines for AI model de...

Design, develop, and optimize data pipelines using Apache NiFi, Spark, and AWS services Manage and maintain AWS RDS (Oracle & Postgres), ensuring high availability and performance tuning Implement data lake and storage solutions using S3 + Iceberg Build real-time data ingestion and streaming pipelines using AWS Kinesis Work with Oracle GoldenGate (OGG) for data replication, CDC, and migration activities Ensure data quality, reliability, and security across all workflows

Position Summary This role involves leading day-to-day technical delivery of data engineering work, especially modernizing existing ETL and scheduling systems. This role focuses on moving data pipelines from Informatica BDM to Databricks and migrating job schedules from TIDAL to Airflow.The person will be hands-on, guide a small team and work closely with architects, analysts and stakeholders on healthcare data platforms. Job Responsibilities Lead migration of existing Informatica BDM ETL jobs to Databricks using PySpark. Migrate job scheduling and dependencies from TIDAL to Apache Airflow. Design, develop and maintain scalable data pipelines on Databricks. Work on healthcare data such as cl...

Position Summary This role involves leading end-to-end delivery of modern data engineering solutions, combining hands-on technical expertise with delivery and team leadership. The role is responsible for designing, building and migrating enterprise-scale data pipelines from legacy platforms (Informatica BDM, TIDAL) to modern cloud-native architectures (Databricks, Apache Airflow), while ensuring data quality, governance and operational stability. This role serves as a critical link between architecture, engineering and delivery teams, ensuring high-quality, timely execution of data initiatives across global programs and orchestrating high-quality data engineering practices with governance ad...

Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increase data accessibility, and fostering data-driven decision making across the organization Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on itWrites unit/integration tests, contributes to engineering wiki, and documents work Designs and develops data integrations and data quality frameworkBachelor of Engineering or equivalent degree in Computer Scienc...

Mandatory Skills AWS Data Services, Azure Data Services, Power BI Reports, DAX Secondary Skills DBT, Airflow, Python We need a Power BI resource with experience in Azure and AWS Cloud migration kind of projects for larger UDP program. Please find below the responsibilities. Understanding Business Requirements Collaborate with stakeholders to understand the business intelligence needs and translate them into technical requirements Data Integration and Modeling Work on integrating data from various sources, ensuring data consistency and accuracy This includes creating and managing data models that support the new AWS environment Report and Dashboard Development Design and develop interactive r...

Work with broader team to build, analyze and improve the AI solutions. You will also work with our software developers in consuming different enterprise applications. Required education Bachelor''s Degree Preferred education Master''s Degree Required technical and professional expertise Resource should have 5-7 years of experience. Sound knowledge of Python and should know how to use the ML related services. Proficient in Python with focus on Data Analytics Packages. Strategy Analyze large, complex data sets and provide actionable insights to inform business decisions. Design and implementing data models that help in identifying patterns and trends. Collaboration Work with data engineers to ...

Lead the end to end testing process for ETL workflows, data pipelines, and data warehouse solutions Collaborate with business analysts, developers, and data engineers to understand data requirements and transformation logic Design and develop comprehensive test strategies, test cases, and test data to validate ETL processes Perform data validation, reconciliation, and verification between source and target systems Implement test automation frameworks for ETL and data validation using tools like Python, PySpark, or SQL based automation Analyze test results, identify defects, and work closely with the development team to ensure timely resolution Monitor and report test progress, metrics, and q...

As an Informatica Data Management Cloud (IDMC) Architect/Tech Lead at Cittabase, you will play a key role in designing, implementing, and maintaining data management solutions using the Informatica Data Management Cloud platform. Your responsibilities will include: - Leading the design and implementation of data management solutions leveraging Informatica Data Management Cloud. - Architecting end-to-end data pipelines for ingestion, transformation, integration, and delivery of data across various sources and destinations. - Collaborating with stakeholders to gather requirements, define data architecture strategies, and translate business needs into technical solutions. - Providing technical ...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering, Hive,Hbase Good to have skills : Map Reduce,Kafka Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensu...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering, HDFS,Hadoop Good to have skills : MapReduce,Hive Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensur...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : Data Science Minimum 2 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data qualit...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering, Hive,Hbase Good to have skills : Snowflake Data Warehouse, Map Reduce,Kafka Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve cre...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 2 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will engage in the design, development, and maintenance of data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating robust data pipelines,...

About The Role Project Role : Technology Architect Project Role Description : Review and integrate all application requirements, including functional, security, integration, performance, quality and operations requirements. Review and integrate the technical architecture requirements. Provide input into final decisions regarding hardware, network products, system software and security. Must have skills : Python (Programming Language) Good to have skills : CommerceTools Commerce Platform Minimum 12 year(s) of experience is required Educational Qualification : 15 years full time education Summary :Define and govern the enterprise GenAI architecture across products and platforms. Own reference ...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Data Engineering Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and imp...