867 Data Pipeline Jobs

As a data infrastructure expert at MPOWER, you will be responsible for designing, building, and maintaining data pipelines in Python and AWS to support the company's business decisions. Your key responsibilities will include: - Designing, building, and maintaining data pipelines in Python and AWS, utilizing services like AWS Redshift, AWS Glue, and MWAA. - Ensuring data quality and integrity through validation, cleansing, and error handling techniques. - Monitoring, troubleshooting, and optimizing the performance of data pipelines for reliability and scalability. - Establishing user needs, monitoring access and security, and capacity planning for database storage requirements. - Building dat...

1.Data Pipeline Devlopment and maintainance 2.Database and Data Warehouse Management 3.data Transformation and Integration https://docs.google.com/spreadsheets/d/1dTIagGAwtsK_iZwlVS1fN3d_ZXfYqyFgjuOba1hhUtc/edit?gid=1273277659#gid=1273277659 Required Candidate profile https://docs.google.com/spreadsheets/d/1dTIagGAwtsK_iZwlVS1fN3d_ZXfYqyFgjuOba1hhUtc/edit?gid=1273277659#gid=1273277659

Data Pipeline Engineer Cyber Data Platform Location: Bangalore Experience: 5–10 years Employment Type: Full-time About the Role Design, build and maintain large-scale, real-time data pipelines for cyber-security events. Stream data to Elasticsearch , Splunk and other downstream platforms. Work closely with global Cyber Security / Incident Response / Threat Investigation teams. Ensure reliability, scalability and high performance across cloud and on-prem environments. Key Responsibilities Build, deploy and manage dataflow tools: Cribl, Logstash, Apache NiFi . Integrate diverse data sources from AWS / Azure / GCP and on-prem systems. Design and implement data mapping, transformation & routing ...

ETL / Pipeline Developer(2) Location-Gurgaon(Onsite) Experience: Minimum 5+ years Immediate joiners Key Responsibilities: Develop, maintain, and optimize ETL processes and data pipelines. Extract, transform, and load data from multiple sources into data warehouse/data lake. Implement data validation, cleansing, and transformation logic. Work with architects to align pipeline development with overall solution design. Monitor performance and troubleshoot ETL processes to ensure smooth data flow. Document ETL workflows and support handover to operations teams. Required Skills: Strong hands-on experience with ETL tools (Talend and similar). Proficiency in SQL, stored procedures, and scripting (P...

Develop and Implement Data Quality Checks: Work with Data Engineers and Business Analysts to define, create, and implement data quality rules and validation checks at various stages of our ETL/ELT pipelines. Monitor and Report on Data Quality: Proactively monitor the health of our data pipelines and datasets. Develop and maintain data quality dashboards (using tools like Power BI or Tableau) to provide clear visibility into data accuracy, completeness, and timeliness. Investigate and Resolve Data Issues: Act as the first line of defense for data-related issues. Investigate data anomalies, perform root cause analysis to identify the source of errors, and collaborate with the engineering team ...

Design and implement an end-to-end data solution architecture tailored to enterprise analytics and operational needs. Build, maintain, and optimize data pipelines and transformations using Python, SQL, and Spark . Manage large-scale data storage and processing with Iceberg, Hadoop, and HDFS . Develop and maintain dbt models to ensure clean, reliable, and well-structured data. Implement robust data ingestion processes , integrating with third-party APIs and on-premise systems. Collaborate with cross-functional teams to align on business goals and technical requirements. Contribute to documentation and continuously improve engineering and data quality processes. Qualifications 4+ years of expe...

Role: Senior Data Engineer Work Location: Hyderabad Experience: 4+ Years Employment Type: Full-Time Work mode : Hybrid Notice period: Immediate - 15 days About the Role We are looking for a highly skilled Data Engineer with 4+ years of experience in building scalable data pipelines, transforming large datasets, and working with modern data platforms. The ideal candidate is proficient in Python , SQL , Apache Spark , and Google Cloud Platform (GCP) , and demonstrates strong communication and collaboration skills. You will play a key role in designing and developing data engineering solutions that enable data-driven decision-making across the organization. Key Responsibilities Design, build, a...

Job Role Overview: We are seeking an experienced Generative AI Solution Architect to join our team. This role requires a deep understanding of AI/ML models, especially generative AI technologies like GPT, DALL•E, Stable Diffusion, etc. The ideal candidate will be responsible for designing, developing, and deploying AI-driven solutions that leverage generative models to solve complex business challenges. You will work closely with cross-functional teams to align AI strategies with business objectives and drive innovation. Key Responsibilities: Strategic AI Solution Design: Collaborate with business leaders and product managers to understand business requirements, pain points, and opportunitie...

Job Summary: As a Data Scientist specializing in Generative AI & Hyper-Personalization, you will work on developing and implementing on use cases related to collections with resolution optimization & machine learning models that power advanced personalization capabilities. You will leverage generative AI techniques to create dynamic, tailored user experiences across multiple touchpoints, while optimizing data-driven strategies to enhance customer-collector/channel engagement, resolution rates, and Low operational cost. Key Responsibilities: Generative AI Model Development: Design, develop, and deploy generative AI models use case (e.g., GPT, GANs, VAEs) to enhance personalization in recommen...

We are seeking an experienced Data Engineer (AI/ML Ops) to design, build, and optimize data pipelines, orchestration frameworks, and data infrastructure supporting large-scale AI/ML programs. This role is pivotal in ensuring seamless data flow, from acquisition and preparation to annotation, validation, and delivery enabling AI/ML models to train on clean, well-structured, and compliant datasets at scale. This role will collaborate with Data Scientists, AI Engineers, IT Operations and Product Teams, working across cloud environments (i.e., AWS, GCP, Azure) and integrating with internal and third-party tools to deliver end-to-end data solutions. The ideal candidate brings a blend of technical...

Job Title:Senior Data Engineer Location: Pune, India Job Type: Full-Time As a Senior Data Engineer at Assent, you will play a critical role in advancing our data engineering practices and serving as a hands-on technical expert. You will contribute significantly to the vision and execution of our data engineering efforts, ensuring the development of secure, robust, scalable, and high-performing data platforms. Your role will involve driving key technical initiatives, collaborating on architectural decisions, and executing complex data engineering tasks to align our data infrastructure with Assent's business goals. You will actively participate in the design, development, and implementation of...

Responsibilities: * Design, develop & maintain data pipelines using Snowflake, Python, SQL & AWS. * Optimize performance & ensure data accuracy through ETL processes.

You are applying for the position of Senior GCP Cloud Administrator for Forbes Advisor, a part of the Forbes Marketplace initiative. As a Senior GCP Cloud Administrator, your role will involve managing and optimizing various aspects of Google Cloud Platform (GCP) to ensure efficient data management and operational practices. **Responsibilities:** - Manage and configure roles/permissions in GCP IAM by following the principle of least privileged access - Optimize Big Query service through slot assignments, SQL Queries, FinOps practices, troubleshooting critical data queries, etc. - Collaborate with various teams for efficient data management and operational practices in GCP - Create automation...

8+ years of Data Engineering experience in a Production cloud infrastructure AWS services - Glue, Redshift, RDS, CloudWatch, CloudFormation, S3, IAM etc. 5+ years experience working in a Big Data ecosystem processing data; includes file systems, data structures/databases, automation, security, messaging, movement, etc. • 6+years in Glue development and 4+ years of experience in development in Python. • Experience in translation of Data mapping documents into PySpark ETL code compatible to be able to run on Glue. • Proficiency in Python and SQL is a must.

Experience: 7+ Years Location: Bangalore Key Responsibilities Design, develop, and maintain scalable ETL pipelines using AWS Glue and PySpark . Lead the data engineering team , ensuring best practices in coding, testing, and deployment. Implement workflow orchestration solutions for complex data pipelines. Optimize data pipelines and queries for advanced SQL performance. Collaborate with data scientists, analysts, and stakeholders to deliver reliable data solutions. Ensure data quality, governance, and security across systems. Drive continuous improvements in ETL processes and architecture. Requirements 7+ years of hands-on experience in Data Engineering . Strong expertise in AWS Glue , ETL ...

Requirements and responsibilities We are seeking a Senior ETL Data Engineer to join our data engineering team supporting compliance and risk initiatives. This is a mission-critical role focused on building scalable, high-performance data pipelines that deliver trusted, high-quality data to business and regulatory stakeholders. What youll do Design, build, and maintain robust, scalable ETL processes for data acquisition, transformation, and delivery; Collaborate with Data Analysts, Architects, and Compliance stakeholders to translate requirements into efficient ETL solutions; Ensure data quality, integrity, lineage, and traceability across large-scale datasets; Optimize ETL workflows for perf...

Key Responsibilities: • Architect, develop, and scale advanced Conversational AI Systems powered by LLMs , RAG , and Agentic AI workflows • Implement Retrieval-Augmented Generation (RAG) pipelines using tools like LangChain , LangGraph , LlamaIndex , and custom retrieval stacks • Engineer agentic behaviors with tools like AutoGen , CrewAI , LangGraph , or custom frameworks for multi-step reasoning and tool usage • Apply prompt engineering , fine-tuning , and LoRA/QLoRA techniques to customize LLMs (e.g., OpenAI GPT-4/5, Mistral, Claude, LLaMA) • Design pipelines to minimize hallucination and toxic output , ensuring factual accuracy, safety, and alignment • Integrate structured (e.g., MongoDB...

Job Description JD Dir/Sr Manager Product Lead Role for Bodhi The Role: As the Dir/Sr Manager Product, you will be a key driver in developing our Agentic AI product suite Collaborating closely with cross-functional teams, you will ensure Director/Senior Manager, Product Agentic AI Are you passionate about AI, data, and building cutting-edge products that drive real impactWere looking for a Director/Senior Manager of Product to lead the development of our Agentic AI product suite This is a unique opportunity to work at the intersection of AI, data, and enterprise solutions, shaping products that enable exponential business outcomes In this role, you will collaborate closely with cross-functio...

This role will be based in Bangalore, India. At LinkedIn, we trust each other to do our best work where it works best for us and our teams. This role offers a hybrid work option, meaning you can both work from home and commute to a LinkedIn office, depending on whats best for you and when it is important for your team to be together. Responsibilities: - Deliver impact by driving innovation while building and shipping software at scale - Provide architectural guidance and mentorship to up-level the engineering organization - Actively improve the level of craftsmanship at LinkedIn by developing best practices and defining best strategies - Design products/services/tools and code that can be us...

This role will be based in Bangalore, India. At LinkedIn, we trust each other to do our best work where it works best for us and our teams. This role offers a hybrid work option, meaning you can both work from home and commute to a LinkedIn office, depending on whats best for you and when it is important for your team to be together. Responsibilities: - Deliver impact by driving innovation while building and shipping software at scale - Provide architectural guidance and mentorship to up-level the engineering organization - Actively improve the level of craftsmanship at LinkedIn by developing best practices and defining best strategies - Design products/services/tools and code that can be us...

This role will be based in Bangalore, India. At LinkedIn, we trust each other to do our best work where it works best for us and our teams. This role offers a hybrid work option, meaning you can both work from home and commute to a LinkedIn office, depending on whats best for you and when it is important for your team to be together. Responsibilities: - Deliver impact by driving innovation while building and shipping software at scale - Provide architectural guidance and mentorship to up-level the engineering organization - Actively improve the level of craftsmanship at LinkedIn by developing best practices and defining best strategies - Design products/services/tools and code that can be us...

This role will be based in Bangalore, India. At LinkedIn, we trust each other to do our best work where it works best for us and our teams. This role offers a hybrid work option, meaning you can both work from home and commute to a LinkedIn office, depending on whats best for you and when it is important for your team to be together. Responsibilities: - Deliver impact by driving innovation while building and shipping software at scale - Provide architectural guidance and mentorship to up-level the engineering organization - Actively improve the level of craftsmanship at LinkedIn by developing best practices and defining best strategies - Design products/services/tools and code that can be us...

Lead Data Engineer Data Management Job description Company Overview Accordion is a global private equity-focused financial consulting firm specializing in driving value creation through services rooted in Data & Analytics and powered by technology. Accordion works at the intersection of Private Equity sponsors and portfolio companies' management teams across every stage of the investment lifecycle. We provide hands-on, execution-oriented support, driving value through the office of the CFO by building data and analytics capabilities and identifying and implementing strategic work, rooted in data and analytics. Accordion is headquartered in New York City with 10 offices worldwide. Join us and...

Design, develop, optimize Azure SQL databases for analytics applications Build maintain ETL/ELT pipelines using Azure Data Factory Azure Databricks for big data processing Leverage Azure data services Synapse Analytics, Cosmos DB, Data Lake Storage Required Candidate profile Implement maintain data warehousing solutions, including data modeling dimensional modeling Ensure data quality, security governance across cloud data platforms Collaborate with cross-functional teams

Project Overview: Offshore Bangalore team is currently engaged in developing real-time ingestion pipelines , pulling timeseries and event frame data from 34+ diverse data sources . This complex data ecosystem requires deep validation strategies, robust automation, and tight collaboration with the Tredence QA team member to ensure high-quality delivery. Key Responsibilities: Data Validation & Quality Strategy: Lead validation of real-time series and event frame data from multiple data sources. Collaborate with Tredence QA to define and implement a robust data QA strategy tailored for real-time ingestion pipelines. Ensure accuracy, completeness, and timeliness of data across the pipeline. Appl...