874 Data Pipeline Jobs - Page 21

Description: Hiring Data Engineer with AWS or GCP Cloud Requirements: Role Summary: The Data Engineer will be responsible for designing, implementing, and maintaining the data infrastructure and pipelines necessary for AI/ML model training and deployment. They will work closely with data scientists and engineers to ensure data is clean, accessible, and efficiently processed Required Experience: • 6-8 years of experience in data engineering, ideally in financial services. • Strong proficiency in SQL, Python, and big data technologies (e.g., Hadoop, Spark). • Experience with cloud platforms (e.g., AWS, Azure, GCP) and data warehousing solutions. • Familiarity with ETL processes and tools. • Kn...

Job Description for the position of Solution Manager At Nielsen, division Media Planning, we develop creative analytical solutions to support companies in the optimization of their marketing and communications budgets. Analytical and econometric approaches form the backbone of these solutions, answering questions like "What media mix works best to communicate my brand message?, What budget should I invest in media and marketing?". Our software tools and other decision-support solutions combine market research, data, modeling results and technical business intelligence. ABOUT THE JOB: In this position, you contribute to the successful implementation of our software solutions for advertisers, ...

Job Description Job Duties and Responsibilities: We are looking for a self-starter to join our Data Engineering team. You will work in a fast-paced environment where you will get an opportunity to build and contribute to the full lifecycle development and maintenance of the data engineering platform. With the Data Engineering team you will get an opportunity to - Design and implement data engineering solutions that is scalable, reliable and secure on the Cloud environment Understand and translate business needs into data engineering solutions Build large scale data pipelines that can handle big data sets using distributed data processing techniques that supports the efforts of the data scien...

Role & responsibilities Proficient: Languages/Framework: Fast API, Azure UI Search API (React) tabases and ETL: Cosmos DB (API for MongoDB), Data Factory Data Bricks Proficiency in Python and R Cloud: Azure Cloud Basics (Azure DevOps) Gitlab: Gitlab Pipeline o Ansible and REX: Rex Deployment Data Science: Prompt Engineering + Modern Testing Data mining and cleaning ML (Supervised/unsupervised learning) NLP techniques, knowledge of Deep Learning techniques include RNN, transformers End-to-end AI solution delivery AI integration and deployment AI frameworks (PyTorch) MLOps frameworks Model deployment processes Data pipeline monitoring Expert: (in addition to proficient skills) Languages/Framew...

Cloud & Data Engineering Specialist Remote Work Contract Duration: 6 months Exp Level: 3 - 4 years (Must be able to work according to JD) Work Timings: (2:30 pm to 11:30 pm IST) We are seeking two highly skilled Cloud & Data Engineering Specialists to join our dynamic team. These roles will focus on designing, building, and optimizing scalable cloud-based solutions, data pipelines, and analytics platforms. The ideal candidates will have strong expertise in cloud platforms, data engineering, and modern technologies, with a focus on delivering robust, secure, and efficient data solutions. Position 1: Cloud & Data Engineering Specialist (Resource 1) Key Responsibilities: Design, develop, and ma...

Job Summary : We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, PySpark, and Power BI, along with a solid background in data integration and the modern Azure ecosystem. The ideal candidate will play a critical role in designing, developing, and implementing scalable data engineering solutions and pipelines. Key Responsibilities : - Design, develop, and implement robust data solutions using Azure Data Factory, Databricks, and related data engineering tools. - Build and maintain scalable ETL/ELT pipelines with a focus on performance and reliability. - Write efficient and reusable code using Python and PySpark. - Perform data cleansing, transformation, and...

Job Title: Senior Data Engineer Location: Remote Job Type: Fulltime YoE: 7 to 10 years relevant experience Shift: 6.30pm to 2.30am IST Job Purpose: The Senior Data Engineer designs, builds, and maintains scalable data pipelines and architectures to support the Denials AI workflow under the guidance of the Team Lead, Data Management. This role ensures data is reliable, compliant with HIPAA, and optimized. Duties & Responsibilities: Collaborate with the Team Lead and crossfunctional teams to gather and refine data requirements for Denials AI solutions. Design, implement, and optimize ETL/ELT pipelines using Python, Dagster, DBT, and AWS data services (Athena, Glue, SQS). Develop and maintain d...

About the Team The Data Platform Tech Team at Navi is instrumental in enabling data-driven decision-making across the organization. We build and manage the core infrastructure and tools required to collect, store, process, and analyze data at scale. Our platforms support self-serve capabilities for both batch and real-time data processing. We work closely with Analytics, Data Science, and Product teams to power a wide range of data use cases across Navi. About the Role As an SDE-3 on the Data Platform team at Navi, you'll design and build large-scale systems powering web events, analytics, and real-time data pipelines. You'll lead backend development, contribute to platform architecture, and...

We are seeking a skilled Data Engineer with expertise in MuleSoft to join our dynamic team In this role, you will be responsible for designing, developing, and maintaining robust data integration solutions that leverage MuleSoft's powerful capabilities You will collaborate closely with cross-functional teams to gather requirements and translate them into scalable data architectures Our ideal candidate is not only proficient in data engineering but also has a strong understanding of API-led connectivity and microservices architecture You will work on various projects that involve data extraction, transformation, and loading (ETL) processes, as well as ensuring the integrity and accessibility ...

-Design, develop & maintain data pipelines using GCP services: Dataflow, BigQuery, and Pub/Sub -Provisioning infrastructure on GCP using IaC with Terraform -Implement & manage data warehouse solutions -Monitor and resolve issues in data workflows Required Candidate profile -Expertise in GCP, Apache Beam, Dataflow, & BigQuery -Pro in Python, SQL, PySpark -Worked with Cloud Composer for orchestration -Solid understanding of DWH, ETL pipelines, and real-time data streaming

ETL / Data Analysis + AWS / Azure Immediate Joiner 5+ years Work From Home Contract to Hire Azure/AWS Spark Terraform Python SQL ETL / Data Pipelines Large Data Handling Data Validation/Transformation Annual bonus Provident fund Health insurance

Who We Are At Kyndryl, we design, build, manage and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward – always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers and our communities. The Role Are you ready to dive headfirst into the captivating world of data engineering at Kyndryl? As a Data Engineer, you'll be the visionary behind our data platforms, crafting them into powerful tools for decision-makers. Your role? Ensuring a treasure trove of pristine, harmonized data is at everyone's fingertips. As a Data Engineer at Kyndryl, you...

Lead Data Engineer Data Management Job description Company Overview Accordion works at the intersection of sponsors and management teams throughout every stage of the investment lifecycle, providing hands-on, execution-focused support to elevate data and analytics capabilities. So, what does it mean to work at Accordion? It means joining 1,000+ analytics, data science, finance & technology experts in a high-growth, agile, and entrepreneurial environment while transforming how portfolio companies drive value. It also means making your mark on Accordions futureby embracing a culture rooted in collaboration and a firm-wide commitment to building something great, together. Headquartered in New Y...

Primine Software Private Limited is looking for Machine Learning Engineer to join our dynamic team and embark on a rewarding career journeyA Machine Learning Engineer is responsible for designing, building, and maintaining systems that use artificial intelligence (AI) and machine learning (ML) algorithms to solve complex problems. 1.Developing and implementing ML algorithms: The Machine Learning Engineer develops and implements machine learning algorithms to solve specific problems, such as natural language processing, computer vision, or predictive modeling.2.Building data pipelines: The Machine Learning Engineer is responsible for building data pipelines that collect, store, and preprocess...

Responsibilities : Collaborate with the sales team to understand customer challenges and business objectives and propose solutions, POC etc. Develop and deliver impactful technical presentations and demos showcasing the capabilities of GCP Data and AI , GenAI Solutions. Conduct technical proof of concepts (POCs) to validate the feasibility and value proposition of GCP solutions. Collaborate with technical specialists and solution architects from COE Team to design and configure tailored cloud solutions. Manage and qualify sales opportunities, working closely with the sales team to progress deals through the sales funnel. Stay up to date on the latest GCP offerings, trends, and best practices...

Exp in dashboard story development, dashboard creation, and data engineering pipelines. Manage and organize large volumes of application log data using Google Big Query Exp with log analytics, user engagement metrics, and product performance metrics Required Candidate profile Exp with tool like Tableau Power BI, or ThoughtSpot AI . Understand log data generated by Python-based applications. Ensure data integrity, consistency, and accessibility for analytical purposes.

Salary: 8 to 24 LPA Exp: 3 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Data...

Salary: 15 to 30 LPA Exp: 3 to 8 years Location : Gurgaon/Bangalore/Pune/Chennai Notice: Immediate to 30 days..!! Key Responsibilities & Skillsets: Common Skillsets : 3+ years of experience in analytics, Pyspark, Python, Spark, SQL and associated data engineering jobs. Must have experience with managing and transforming big data sets using pyspark, spark-scala, Numpy pandas Excellent communication & presentation skills Experience in managing Python codes and collaborating with customer on model evolution Good knowledge of data base management and Hadoop/Spark, SQL, HIVE, Python (expertise). Superior analytical and problem solving skills Should be able to work on a problem independently and p...

Role & responsibilities The individual will be responsible for designs, builds, and maintains scalable data solutions including data models, ETL pipelines, and real-time processing systems He/she is also responsible for mentoring junior team members and play a pivotal role in overseeing the technical aspects of data related projects. MUST HAVE Experience on working with ETL on a data warehouse or big data projects. Experience with cloud services like Azure or GCP or AWS with respect to the data related services, could be just ETL ,cleanup or BI but has potential to be groomed in a 1-2 month period. Should have energy and enthusiasm to get into new things, able to stretch. Good SQL skills and...

Role & responsibilities What You'll Do Build the underneath data platform and maintain data processing pipelines using best in class technologies. Special focus on R&D to challenge status-quo and build the next generation data mesh that is efficient and cost effective. Translate complex technical and functional requirements into detailed designs. Who You Are Strong programming skills (Python, Java, and Scala) Experience writing SQL , structuring data, and data storage practices Experience with data modeling Knowledge of data warehousing concepts Experienced building data pipelines and micro services Experience with Spark , Airflow and other streaming technologies to process incredible volume...

Key Responsibilities: - Architect and Develop GPT Models:** Design, build, and optimize custom GPT models, including architecture and fine-tuning, to meet the specific needs of the legal domain. - Cloud Architecture: Develop and manage cloud-based infrastructure with a deep understanding of cloud services and architecture to deploy and scale AI solutions efficiently. - Integration and Deployment: Oversee the deployment of GenAI models and ensure seamless integration with the KonProz platform. - Vector DB Utilization: Work with Vector DBs to enhance data retrieval, storage, and interaction with AI models. - Performance Optimization: Monitor and optimize the performance of AI models and cloud ...

Company Overview We are a fast-growing research firm with a global footprint, addressing major market, strategic, economic, scientific, and technological developments for business leaders in industrial, pharmaceutical, and technology organizations. We provides comprehensive analysis of global market sizing, forecasting, and industry intelligence, covering markets where advances in science and technology are improving the quality, standard, and sustainability of businesses, economies, and lives. We are in the process of retooling the business for next-generation innovative online and AI-powered products. Job Summary We are seeking a skilled Agentic AI Technical Engineer to design, develop, an...

Experience: 4-8 Years Job Location: Remote No. of Position: Multiple Qualifications: B Tech / M Tech/ MCA or Higher Work Timings: 1:30 PM IST to 10:30 PM IST Functional Area: Data Engineering Job Description: We are seeking a skilled Data Engineer with 4 to 8 years of experience to join our team. The ideal candidate will have a strong background in Python programming, along with expertise in AWS or Azure services. The candidate should also possess solid SQL skills and be proficient in web scraping techniques. Role and responsibilities: Develop and maintain data pipelines using Python, PySpark, and SQL to extract, transform, and load data from various sources. Implement and optimize data proc...

Overview We are looking for a Senior Data Engineer with deep hands-on expertise in PySpark, Databricks, and distributed data architecture. This individual will play a lead role in designing, developing, and optimizing data pipelines critical to our Ratings Modernization, Corrections, and Regulatory implementation programs under PDB 2.0. The ideal candidate will thrive in fast-paced, ambiguous environments and collaborate closely with engineering, product, and governance teams. Responsibilities Design, develop, and maintain robust ETL/ELT pipelines using PySpark and Databricks . Own pipeline architecture and drive performance improvements through partitioning, indexing, and Spark optimization...

5-9yrs exp in Pipeline Monitoring, working data, ETL, Data warehouse & BI Projects. Coding exp, Develop & maintain Dashboards. Knowledge in AWS Glue, S3, Athena, JIRA, Programming- Python, Java, Scala etc. Bangalore/ Pune/ Hyderabad/ Greater Noida.