874 Data Pipeline Jobs - Page 18

About the job : Role : Microsoft Fabric Data Engineer Experience : 6+ years as Azure Data Engineer including at least 1 E2E Implementation in Microsoft Fabric. Responsibilities : - Lead the design and implementation of Microsoft Fabric-centric data platforms and data warehouses. - Develop and optimize ETL/ELT processes within the Microsoft Azure ecosystem, effectively utilizing relevant Fabric solutions. - Ensure data integrity, quality, and governance throughout Microsoft Fabric environment. - Collaborate with stakeholders to translate business needs into actionable data solutions. - Troubleshoot and optimize existing Fabric implementations for enhanced performance. Skills : - Solid foundat...

Hi Candidates, we have job openings in one of our MNC Company Interested candidates can apply here and share details to chandrakala.c@i-q.co Note: NP-0-15 days only serving Role & responsibilities Key Responsibilities Complete Data Modelling Tasks o Initiate and manage Gap Analysis and Source-to-Target Mapping Exercises. o Gain a comprehensive understanding of the EA extract. o Map the SAP source used in EA extracts to the AWS Transform Zone, AWS Conform Zone, and AWS Enrich Zone. Develop a matrix view of all Excel/Tableau reports to identify any missing fields or tables from SAP in the Transform Zone. Engage with SMEs to finalize the Data Model (DM). Obtain email confirmation and approval f...

Role Description: As a Data Engineering Lead, you will play a crucial role in overseeing the design, development, and maintenance of our organization's data architecture and infrastructure. You will be responsible for designing and developing the architecture for the data platform that ensures the efficient and effective processing of large volumes of data, enabling the business to make informed decisions based on reliable and high-quality data. The ideal candidate will have a strong background in data engineering, excellent leadership skills, and a proven track record of successfully managing complex data projects. Responsibilities : Data Architecture and Design: Design and implement scalab...

Job Title: Data Engineer Location: Pune Experience: Fresher 5 years Employment Type: Full-Time About the Role: We are looking for talented Data Engineers to join our expanding team. You will be responsible for integrating, managing, and optimizing data systems thus contributing to innovative and challenging projects. This is a great opportunity to enhance your technical skills and become a part of a successful team. Key Responsibilities: Develop, customize, and manage data integration tools, databases (MySQL), and data warehouses using Python, Java, and other relevant technologies. Write and execute complex queries and automation scripts for processing operational data. Collaborate with the ...

Qualification BCA. MCA preferable Required Skill Set 5+ years in Data Engineering, with at least 2 years on GCP/BigQuery Strong Python and SQL expertise (Airflow, dbt or similar) Deep understanding of ETL patterns, change-data-capture, and data-quality frameworks Experience with IoT or time-series data pipelines a plus Excellent communication skills and track record of leading cross-functional teams Job Description / Responsibilities Design, build, and maintain scalable ETL/ELT pipelines in Airflow and BigQuery Define and enforce data-modeling standards, naming conventions, and testing frameworks Develop and review core transformations: IoT enrichment (batch-ID assignment, stage tagging) Tra...

The successful candidate for the Full Stack Developer position at U.S. Pharmacopeial Convention (USP) will have a demonstrated understanding of the organization's mission and a commitment to excellence through inclusive and equitable behaviors and practices. They should possess the ability to quickly build credibility with stakeholders. As a Full Stack Developer, you will be part of the Digital & Innovation group at USP, responsible for building innovative digital products using cutting-edge cloud technologies. Your role will be crucial in creating an amazing digital experience for customers. Your responsibilities will include building scalable applications and platforms using the latest clo...

As a Data Engineer2 at GoKwik, you will have the opportunity to closely collaborate with product managers, data scientists, business intelligence teams, and SDEs to develop and implement data-driven strategies. Your role will involve identifying, designing, and executing process improvements to enhance data models, architectures, pipelines, and applications. You will play a vital role in continuously optimizing data processes, overseeing data management, governance, security, and analysis to ensure data quality and security across all product verticals. Additionally, you will design, create, and deploy new data models and pipelines as necessary to achieve high performance, operational excell...

We are looking for someone who is enthusiastic to contribute to the implementation of a metadata-driven platform managing the full lifecycle of batch and streaming Big Data pipelines. This role involves applying ML and AI techniques in data management, such as anomaly detection for identifying and resolving data quality issues and data discovery. The platform facilitates the delivery of Visa's core data assets to both internal and external customers. You will provide Platform-as-a-Service offerings that are easy to consume, scalable, secure, and reliable using open source-based Cloud solutions for Big Data technologies. Working at the intersection of infrastructure and software engineering, ...

Position summary: We are seeking a Senior Software Development Engineer – Data Engineering with 3-5 years of experience to design, develop, and optimize data pipelines and analytics workflows using Snowflake, Databricks, and Apache Spark. The ideal candidate will have a strong background in big data processing, cloud data platforms, and performance optimization to enable scalable data-driven solutions. Key Responsibilities: Work with cloud-based data solutions (Azure, AWS, GCP). Implement data modeling and warehousing solutions. Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes. Designing and optimizing data storage solutions...

Role & responsibilities Design and implement AI agent workflows. Develop end-to-end intelligent pipelines and multi-agent systems (e.g., LangGraph/LangChain workflows) that coordinate multiple LLM-powered agents to solve complex tasks. Create graph-based or state-machine architectures for AI agents, chaining prompts and tools as needed. Build and fine-tune generative models. Develop, train, and fine-tune advanced generative models (transformers, diffusion models, VAEs, GANs, etc.) on domain-specific data. Deploy and optimize foundation models (such as GPT, LLaMA, Mistral) in production, adapting them to our use cases through prompt engineering and supervised fine-tuning. Develop data pipelin...

As a Mid Databricks Engineer, you will play a pivotal role in designing, implementing, and optimizing data processing pipelines and analytics solutions on the Databricks platform. You will collaborate closely with cross-functional teams to understand business requirements, architect scalable solutions, and ensure the reliability and performance of our data infrastructure. This role requires deep expertise in Databricks, strong programming skills, and a passion for solving complex engineering challenges. What you'll do : - Design and develop data processing pipelines and analytics solutions using Databricks. - Architect scalable and efficient data models and storage solutions on the Databrick...

Key Responsibilities Design, develop, and maintain scalable ETL workflows using Cloud Data Fusion and Apache Airflow . Configure and manage various data connectors (e.g., Cloud Storage, Pub/Sub, JDBC, SaaS APIs) for batch and streaming data ingestion. Implement data transformations, cleansing, and enrichment logic in Python (and SQL) to meet analytic requirements. Optimize BigQuery data models (fact/dimension tables, partitioning, clustering) for performance and cost-efficiency. Monitor, troubleshoot, and tune pipeline performance; implement robust error-handling and alerting mechanisms. Collaborate with data analysts, BI developers, and architects to understand data requirements and deliver...

Position: Senior Data Engineer Location: Bangalore, India About Dodge Dodge Construction Network exists to deliver the comprehensive data and connections the construction industry needs to build thriving communities. Our legacy is deeply rooted in empowering our customers with transformative insights, igniting their journey towards unparalleled business expansion and success. We serve decision-makers who seek reliable growth and who value relationships built on trust and quality. By combining our proprietary data with cutting-edge software, we deliver to our customers the essential intelligence needed to excel within their respective landscapes. We propel the construction industry forward by...

Role & Apache Nifi 5+ years of hands-on experience with Apache NiFi, including developing, managing, and optimizing complex data flows in production environments. Proven experience with Cloudera NiFi (CDP Data Flow) in enterprise environments, including integration with Cloudera Manager. Experience migrating NiFi flows across major version upgrades with strong understanding of backward compatibility Strong proficiency in Groovy scripting, used for ExecuteScript and InvokeScriptedProcessor processors. Solid understanding of SSH and SFTP protocols, including authentication schemes (key-based, password), session negotiation, and file permissions handling in NiFi processors (e.g., ListSFTP, Fetc...

Job Role: Snowflake / SQL Architect Duration: 6 months contract Job Description: • Architect and manage scalable data solutions using Snowflake and advanced SQL, optimizing performance for analytics and reporting. • Design and implement data pipelines, data warehouses, and data lakes, ensuring efficient data ingestion and transformation. • Develop best practices for data security, access control, and compliance within cloud-based data environmentsCollaborate with cross-functional teams to understand business needs and translate them into robust data architectures. • Evaluate and integrate third-party tools and technologies to enhance the Snowflake ecosystem and overall data strategy.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

An Opportunity to Work with One of India's Leading Credit Card Tech Innovators BOBCARD (A Bank of Baroda Subsidiary) Educational Qualifications: Bachelors/Masters degree in Computer Science, Data Engineering, Information Technology, or a related field . Experience Requirement: 5+ years Applicants should possess the following attributes: Technical Skills: Should have knowledge database design and development skills in Non- Relational Databases, SQL /. PL SQL, Oracle/MS SQL/MySQL Following GCP skillsets are necessary - BigQuery Storage, BigQuery ML, Cloud Data Fusion, Looker Studio Pro, Looker ML, DataStream Google Cloud platform especially Big Query Dataflow Experience with Java and Python an...

Full time with top German MNC for location Bangalore - Experience on SCALA is a must Job Overview: To work on development, monitoring and maintenance of Data pipelines across clusters. Primary responsibilities: Develop, Monitor and Maintain data pipeline for various plants. Create and maintain optimal data pipeline architecture. Assemble large, complex data sets that meet functional / non-functional business requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability. Work with stakeholders including the Data officers and stewards to assist with data-related technical i...

About Agoda Agoda is an online travel booking platform that offers accommodations, flights, and more to travelers worldwide. With a global network of 4.7M hotels and holiday properties, as well as flights, activities, and more, we are dedicated to connecting travelers with seamless travel experiences. As part of Booking Holdings and based in Asia, our team of 7,100+ employees from 95+ nationalities across 27 markets creates a work environment that thrives on diversity, creativity, and collaboration. At Agoda, we foster a culture of innovation through experimentation and ownership, allowing our customers to explore and enjoy the world. Our Purpose: Bridging the World Through Travel We believe...

- Own and enhance the full lifecycle of our data pipelines: from ingestion to transformation to delivery - Design, implement, and scale robust data infrastructure in a cloud environment - Write high-quality, production-grade code using Python

Job Title: Senior Data Engineer Python, PySpark, AWS Experience Required: 6 to 12 Years Location: Hyderabad Job Type: Full Time / Permanent Job Description: We are looking for a passionate and experienced Senior Data Engineer to join our team in Hyderabad . The ideal candidate should have a strong background in data engineering on AWS , with hands-on expertise in Python, PySpark, and AWS services to build and maintain scalable data pipelines and ETL workflows. Mandatory Skills: Data Engineering Python PySpark AWS Services (S3, Glue, Lambda, Redshift, RDS, EC2, Data Pipeline) Key Responsibilities: Design and implement robust, scalable data pipelines using PySpark , AWS Glue , and AWS Data Pip...

When our values align, there's no limit to what we can achieve. At Parexel, we all share the same goal - to improve the world's health. From clinical trials to regulatory, consulting, and market access, every clinical development solution we provide is underpinned by something special - a deep conviction in what we do. Each of us, no matter what we do at Parexel, contributes to the development of a therapy that ultimately will benefit a patient. We take our work personally, we do it with empathy and we're committed to making a difference. Required Skills: With a focus on automating testing and DevOps, use Microsoft Azure data PaaS services, design, build, modify, and support data pipelines l...

Were looking for a Senior Data Analyst to join our data-driven team at an ad-tech company that thrives on turning complexity into clarity. Our analysts play a critical role in transforming raw, noisy data into accurate, actionable signals that drive real-time decision-making and long-term strategy. Youll work closely with product, engineering, and business teams to uncover insights, shape KPIs, and guide performance optimization. Responsibilities: Analyze large-scale datasets from multiple sources to uncover actionable insights and drive business impact. Design, monitor, and maintain key performance indicators (KPIs) across ad delivery, bidding, and monetization systems. Partner with product...

About the job : - As a Mid Databricks Engineer, you will play a pivotal role in designing, implementing, and optimizing data processing pipelines and analytics solutions on the Databricks platform. - You will collaborate closely with cross-functional teams to understand business requirements, architect scalable solutions, and ensure the reliability and performance of our data infrastructure. - This role requires deep expertise in Databricks, strong programming skills, and a passion for solving complex engineering challenges. What You'll Do : - Design and develop data processing pipelines and analytics solutions using Databricks. - Architect scalable and efficient data models and storage solu...

Role Responsibilities : - Design and implement data pipelines using MS Fabric. - Develop data models to support business intelligence and analytics. - Manage and optimize ETL processes for data extraction, transformation, and loading. - Collaborate with cross-functional teams to gather and define data requirements. - Ensure data quality and integrity in all data processes. - Implement best practices for data management, storage, and processing. - Conduct performance tuning for data storage and retrieval for enhanced efficiency. - Generate and maintain documentation for data architecture and data flow. - Participate in troubleshooting data-related issues and implement solutions. - Monitor and...