3177 Data Pipelines Jobs - Page 13

ABOUT THE ROLE Role Description: This role is responsible for designing, developing, and implementing automation solutions to support Anaplan application workflows and data pipeline validation . The Data Engineer will leverage Python , Pytest , and Selenium frameworks to automate Anaplan functional processes, and will also design and maintain an ETL test framework to ensure the quality, integrity, and accuracy of data across systems. The engineer will work closely with data engineering, QA, and application teams to integrate automated testing into CI/CD pipelines, enhance test efficiency, and drive continuous improvement in data quality and system reliability. Primary Roles & Responsibilitie...

We are looking for a highly skilled Staff AI Engineer to join our team at Rapid7, contributing to the development of innovative artificial intelligence solutions. The ideal candidate will have a strong background in AI engineering and a passion for working with cutting-edge technologies. Roles and Responsibility Design, develop, and deploy scalable AI models and algorithms to solve complex problems. Collaborate with cross-functional teams to identify business needs and develop AI-powered solutions. Develop and maintain large-scale data pipelines and architectures to support AI model development. Conduct research and stay up-to-date with industry trends and advancements in AI technology. Work...

We are looking for a skilled AI Developer to join our team at Mindtech Solutions, playing a crucial role in developing innovative artificial intelligence solutions. The ideal candidate will have a strong foundation in AI and machine learning, with the ability to design, develop, and deploy AI models. Roles and Responsibility Design and develop scalable AI and machine learning models using various algorithms and techniques. Collaborate with cross-functional teams to identify business problems and propose AI solutions. Develop and maintain large-scale data pipelines and architectures to support AI model development. Implement automated testing and deployment scripts to ensure efficient model d...

We are looking for a highly skilled Senior AI Engineer (Data Science) with 5 to 10 years of experience to join our team. The ideal candidate will have a strong background in data science and artificial intelligence, with excellent problem-solving skills. Roles and Responsibility Design and develop scalable data pipelines using machine learning algorithms. Collaborate with cross-functional teams to integrate AI models into existing systems. Develop and maintain large-scale data warehouses and data lakes. Implement predictive modeling techniques to drive business growth. Work closely with stakeholders to identify business problems and develop solutions. Stay up-to-date with industry trends and...

We are looking for a highly skilled and experienced Gen AI Architect to join our team at Myridius x Aethereus, with a strong background in artificial intelligence and machine learning. The ideal candidate will have 10-15 years of experience. Roles and Responsibility Design and develop scalable and efficient AI architectures to solve complex problems. Collaborate with cross-functional teams to integrate AI models into existing systems. Develop and maintain large-scale data pipelines and architectures. Conduct research and stay updated on the latest advancements in AI and machine learning. Lead the development of new AI products and features. Work closely with stakeholders to understand busine...

We are looking for passionate and skilled Software Developers to join our Core AI Services team. In this role, you will help design, develop, and scale AI-enabling platform services and public APIs that are secure, reliable, and cloud-native. These services will act as foundational building blocks for AI adoption across AVEVAs product portfolio and partner ecosystem. You will part of a Scrum team to build innovative, standards-compliant, secure and production-grade AI capabilities, with a builder mindset - rapid prototyping and continuous improvement with agility of a start-up. Key Responsibilities: Build scalable, fault-tolerant cloud-native services on Microsoft Azure, ensuring high perfor...

Title and Summary Software Engineer II (Bigdata / Spark / Apache / NIFI/ AirFlow / Unix / Oracle / Hadoop)OverviewMastercard is the global technology company behind the worlds fastest payments processing network We are a vehicle for commerce, a connection to financial systems for the previously excluded, a technology innovation lab, and the home of Priceless We ensure every employee has the opportunity to be a part of something bigger and to change lives We believe as our company grows, so should you We believe in connecting everyone to endless, priceless possibilities As a Test Engineer in Data Platform and Engineering Services, you will have the opportunity to build frameworks to perform t...

Role Description: In this vital role you will be responsible for the development and implementation of our data strategy. The ideal candidate possesses a strong blend of technical expertise and data-driven problem-solving skills. As a Data Engineer, you will play a crucial role in building, and optimizing our data pipelines and platforms in a SAFE Agile product team. The Senior Data Engineer will design, develop, and maintain data engineering applications and solutions that meet business needs and ensuring the availability and performance of critical systems and applications. This role involves working closely with product managers, designers, and other engineers to create high-quality, scal...

We are looking for a Technical Architect, who can work closely with SAP and analytics teams to design and build scalable data solutions. The role involves working on graph-based data models, ML-oriented data pipelines, and enterprise integrations. Required Candidate profile experience with graph databases, preferably Neo4j Strong in data modeling and schema design for ML-driven applications.Proficient in AWS services,API development and microsoft teams

Design, develop, and maintain scalable data pipelines using BigQuery, DBT, and PythonImplement and optimize ETL/ELT workflows on Google Cloud Platform (GCP) using services like Cloud Composer, Dataflow, and Cloud Storage Provide production support for data workflows, ensuring high availability, performance, and data integrityMonitor and troubleshoot issues in batch and streaming pipelines Collaborate with cross-functional teams to gather requirements and deliver robust data solutionsAutomate operational tasks and build alerting mechanisms using Python scripts and GCP monitoring tools Maintain and enhance DBT models, ensuring modularity, test coverage, and documentationEnsure compliance with ...

PURPOSE AND SCOPE : The Business Analyst provides health operations and business systems expertise during the receipt, review and delivery of client requests and internal projects related to data integrations, reporting and analytics for the claims administration solution. PRINCIPAL RESPONSIBILITIES AND DUTIES: Manage resolution of incoming client tickets, including gathering information, recreation of issues, resolution of the issue and co-ordination with developers Analyze incoming projects, client and strategic, requests and gather business requirements Ensure appropriate communication with account management and clients on all projects and tickets Execute QA test plans and co-ordinate is...

Strong experience as an Azure Cloud Architect with hands-on PySpark, Databricks, SQL, and ADF Proven experience in designing end-to-end cloud data architecture Hands-on expertise in data pipelines, ETL/ELT, and analytics solutions Experience in leading and managing teams for cloud/data projects Collaborate with business and technical stakeholders to translate requirements Ensure data quality, security, scalability, and performance of solutions Strong knowledge of Azure ecosystem and cloud best practices Ability to provide technical leadership, mentorship, and architectural guidance Strong problem-solving, analytical, and communication skills Experience in implementing cloud governance and st...

Job Highlights Design, build, and operate secure, transparent data pipelinesmoving and transforming data fromSAP BW/ERP, OSIsoft PI, Dynamics CRM, web APIs, SFTP, and file shares to drive analytics and AI initiatives. Develop scalable batch and streaming processes using PySpark, working in environments like Azure Synapse. Build, test, and deploy CI/CD workflows for data pipelines (using Git, Azure DevOps, DataLake, ADF, Synapse Pipelines). Maintain documentation, unit tests, and code reviews for robustness and maintainability. Harmonize different data formats and sourcesincluding Parquet, CSV, JSON, XML, time series, andclassic relational structures. Set up robust security structures (user/r...

Overview : Transfer Solutions is responsible for driving Mastercards expansion into new payment flows such as Disbursements & Remittances. The team is working on creating a market-leading money transfer proposition, Mastercard Move, to power the next generation of payments between people and businesses, whether money is moving domestically or across borders, by delivering the ability to pay and get paid with choice, transparency, and flexibility. The Product & Engineering teams within Transfer Solutions are responsible for designing, developing, launching, and maintaining products and services designed to capture these flows from a wide range of customer segments. By addressing customer pain...

Job Title : Senior Databricks Developer Position : SSE / LA Experience : 5 to 10 years of experience Category : Software Development Work Type : Hybrid Employment Type : Full Time / Permanent Qualification : Bachelors or Masters degree in Computer Science, Engineering, or a related field. Role Summary Databricks with strong data engineering skills and deep AWS cloud expertise. To lead the design, build, and deployment of data products on the Databricks Lakehouse platform, ensuring scalability, governance, security, and business value delivery. The role requires mastery in Databricks features, Delta Lake, AWS ecosystem integration, and modern Data Engineering best practices. Primary Responsib...

We are building a customer-facing data platform from the ground up. The platform requires real-time data ingestion, high availability, and integration with Generative AI. You will lead architecture decisions on Azure, sitting between Backend Engineering and Data Engineering. You will own a code-first approach: Python-based pipelines and APIs, not drag-and-drop ETL tools. Key Responsibilities Design event-driven ingestion architectures. Select appropriate Azure compute (Functions, Durable Functions, containers) based on workload characteristics. Design OLTP schemas for sub-second application retrieval, not just offline reporting. Architect data flows for RAG workflows. Ensure unstructured tex...

Job Summary : You will be responsible for designing, building, and maintaining our data infrastructure. This role involves working closely with data scientists, developers, and other stakeholders to ensure data availability, quality, and reliability. Key Responsibilities : Design, develop, and maintain scalable data pipelines to collect, process, and store large volumes of data from various sources. Implement ETL (Extract, Transform, Load) processes to transform raw data into structured formats. Integrate data from multiple sources, including databases, APIs, third-party services, and data warehouses. Ensure data consistency and integrity across different data stores. Design and manage datab...

Project description We are looking for a skilled Data Engineer with strong experience in AWS and Apache Airflow to join a dynamic data team. You will be responsible for building and maintaining scalable data pipelines, orchestrating workflows, and ensuring data quality and availability across platforms. Responsibilities Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient ETL/ELT pipelines for ingesting, transforming, and loading data from various sources into our data lake and data warehouse. AWS Expertise: Develop and manage data solutions using a wide range of AWS services, including but not limited to: Storage: S3 (for data lakes), RDS (PostgreSQL, MySQL...

Required Skills & Experience -3+ years in data engineering or AI infrastructure roles -Expertise in AWS or Azure (not required to know both) -Hands-on experience with vector stores and embedding pipelines -Strong Python development experience -Experience with OCR/document intelligence tools -Strong familiarity with LLMs, RAG architectures, embeddings, and retrieval techniques -Experience with CICD and software engineering best practices -Experience building and maintaining data pipelines Job Description Insight Global is seeking an AI Data Engineer with deep expertise in AWS or Azure to join one of our medical device clients local to India. In this role, you will develop scalable ETL/ELT pip...

About The Role Project Role : AI / ML Engineer Project Role Description : Develops applications and systems that utilize AI tools, Cloud AI services, with proper cloud or on-prem application pipeline with production ready quality. Be able to apply GenAI models as part of the solution. Could also include but not limited to deep learning, neural networks, chatbots, image processing. Must have skills : Large Language Models Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education SUMMARY:We are looking for a technical enthusiast with strong expertise in Python and a solid understanding of AI/ML concepts to contribute to the de...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform * Responsibilities: * Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS * Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform * Experience in d...

Work with broader team to build, analyze and improve the AI solutions. You will also work with our software developers in consuming different enterprise applications Required education Bachelor's Degree Required technical and professional expertise Resource should have 5-7 years of experience. Sound knowledge of Python and should know how to use the ML related services. Proficient in Python with focus on Data Analytics Packages. Strategy Analyse large, complex data sets and provide actionable insights to inform business decisions. Strategy Design and implementing data models that help in identifying patterns and trends. Collaboration Work with data engineers to optimize and maintain data pip...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and Azure Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark and Hive, Hbase or other NoSQL databases on Azure Cloud Data Platform or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing strea...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...