1175 Data Pipeline Jobs - Page 13

Looking for Data Engineer / Big Date with Python ,ETL, SQL, and data Pipeline - Contract to hire

8+ years of experience in application maintenance and support, with a focus on data-driven systems. Experience of managing Cloud Data based applications (preferably Snowflake, Matillion or other ETL tools) especially from Support & Maintenance perspective. Experience with ITSM tools (e.g., ServiceNow, Jira) for tracking Service Requests and enhancements. Performing Root Cause analysis and taking steps to reduce recurring incidents. Collaborate with cross-functional teams including Data Engineers, Data Architects and business analysts to ensure seamless application operations. Manage application upgrades, patches, and configuration changes. Develop and maintain documentation for application a...

Executes standard software solutions, design, development, and technical troubleshooting Writes secure and high-quality code using the syntax of at least one programming language with limited guidance Designs, develops, codes, and troubleshoots with consideration of upstream and downstream systems and technical implications Applies knowledge of tools within the Software Development Life Cycle toolchain to improve the value realized by automation Applies technical troubleshooting to break down solutions and solve technical problems of basic complexity Gathers, analyzes, and draws conclusions from large, diverse data sets to identify problems and contribute to decision-making in service of sec...

Who We Are At Kyndryl, we design, build, manage and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward – always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers and our communities. The Role Are you ready to embark on an exhilarating journey as a Data Consultant? Join Kyndryl and become a driving force behind the transformative power of data! We're seeking an exceptionally talented individual to accelerate the competitive performance of our customers worldwide, establishing us as their unrivaled business and technology consulting p...

• Proficient in Git for version control , SQL & exp with DB technologies,data pipeline tools such as Apache Airflow,data migration and validation techniques.Strong knowledge of Apache Spark for data processing and transformation,data governance.

Data & AI Data Platform Engineer Description: Be a part of our success story. Launch offers talented and motivated people the opportunity to do the best work of their lives in a dynamic and growing company. Through competitive salaries, outstanding benefits, internal advancement opportunities, and recognized community involvement, you will have the chance to create a career you can be proud of. Your new trajectory starts here at Launch. What are we looking for: As a Data Platform Engineer, you will be instrumental in assisting with the blueprint and design of the organization's data platform, covering all relevant components. Your daily activities will include close collaboration with Integr...

Key responsibilities include: Data Architecture: Perform all technical aspects of data architecture and database management for ID&A, including developing data pipelines, new database structures and APIs as applicable Data Design: Translate logical data architectures into physical data designs, ensuring alignment with data modeling best practices and standards Data Process and Monitoring: Ensure proper data ingestion, validation, testing, and monitoring for ID&A data lake Support database and data platform administration for initiatives building, enhancing, or maintaining databases, data warehouses and data pipelines Data Migration: Design and support migration to a data model that is indepe...

Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes: Act creatively ...

As a data infrastructure expert at MPOWER, you will be responsible for designing, building, and maintaining data pipelines in Python and AWS to support the company's business decisions. Your key responsibilities will include: - Designing, building, and maintaining data pipelines in Python and AWS, utilizing services like AWS Redshift, AWS Glue, and MWAA. - Ensuring data quality and integrity through validation, cleansing, and error handling techniques. - Monitoring, troubleshooting, and optimizing the performance of data pipelines for reliability and scalability. - Establishing user needs, monitoring access and security, and capacity planning for database storage requirements. - Building dat...

1.Data Pipeline Devlopment and maintainance 2.Database and Data Warehouse Management 3.data Transformation and Integration https://docs.google.com/spreadsheets/d/1dTIagGAwtsK_iZwlVS1fN3d_ZXfYqyFgjuOba1hhUtc/edit?gid=1273277659#gid=1273277659 Required Candidate profile https://docs.google.com/spreadsheets/d/1dTIagGAwtsK_iZwlVS1fN3d_ZXfYqyFgjuOba1hhUtc/edit?gid=1273277659#gid=1273277659

Data Pipeline Engineer Cyber Data Platform Location: Bangalore Experience: 5–10 years Employment Type: Full-time About the Role Design, build and maintain large-scale, real-time data pipelines for cyber-security events. Stream data to Elasticsearch , Splunk and other downstream platforms. Work closely with global Cyber Security / Incident Response / Threat Investigation teams. Ensure reliability, scalability and high performance across cloud and on-prem environments. Key Responsibilities Build, deploy and manage dataflow tools: Cribl, Logstash, Apache NiFi . Integrate diverse data sources from AWS / Azure / GCP and on-prem systems. Design and implement data mapping, transformation & routing ...

ETL / Pipeline Developer(2) Location-Gurgaon(Onsite) Experience: Minimum 5+ years Immediate joiners Key Responsibilities: Develop, maintain, and optimize ETL processes and data pipelines. Extract, transform, and load data from multiple sources into data warehouse/data lake. Implement data validation, cleansing, and transformation logic. Work with architects to align pipeline development with overall solution design. Monitor performance and troubleshoot ETL processes to ensure smooth data flow. Document ETL workflows and support handover to operations teams. Required Skills: Strong hands-on experience with ETL tools (Talend and similar). Proficiency in SQL, stored procedures, and scripting (P...

Develop and Implement Data Quality Checks: Work with Data Engineers and Business Analysts to define, create, and implement data quality rules and validation checks at various stages of our ETL/ELT pipelines. Monitor and Report on Data Quality: Proactively monitor the health of our data pipelines and datasets. Develop and maintain data quality dashboards (using tools like Power BI or Tableau) to provide clear visibility into data accuracy, completeness, and timeliness. Investigate and Resolve Data Issues: Act as the first line of defense for data-related issues. Investigate data anomalies, perform root cause analysis to identify the source of errors, and collaborate with the engineering team ...

Design and implement an end-to-end data solution architecture tailored to enterprise analytics and operational needs. Build, maintain, and optimize data pipelines and transformations using Python, SQL, and Spark . Manage large-scale data storage and processing with Iceberg, Hadoop, and HDFS . Develop and maintain dbt models to ensure clean, reliable, and well-structured data. Implement robust data ingestion processes , integrating with third-party APIs and on-premise systems. Collaborate with cross-functional teams to align on business goals and technical requirements. Contribute to documentation and continuously improve engineering and data quality processes. Qualifications 4+ years of expe...

Role: Senior Data Engineer Work Location: Hyderabad Experience: 4+ Years Employment Type: Full-Time Work mode : Hybrid Notice period: Immediate - 15 days About the Role We are looking for a highly skilled Data Engineer with 4+ years of experience in building scalable data pipelines, transforming large datasets, and working with modern data platforms. The ideal candidate is proficient in Python , SQL , Apache Spark , and Google Cloud Platform (GCP) , and demonstrates strong communication and collaboration skills. You will play a key role in designing and developing data engineering solutions that enable data-driven decision-making across the organization. Key Responsibilities Design, build, a...

Job Role Overview: We are seeking an experienced Generative AI Solution Architect to join our team. This role requires a deep understanding of AI/ML models, especially generative AI technologies like GPT, DALL•E, Stable Diffusion, etc. The ideal candidate will be responsible for designing, developing, and deploying AI-driven solutions that leverage generative models to solve complex business challenges. You will work closely with cross-functional teams to align AI strategies with business objectives and drive innovation. Key Responsibilities: Strategic AI Solution Design: Collaborate with business leaders and product managers to understand business requirements, pain points, and opportunitie...

Job Summary: As a Data Scientist specializing in Generative AI & Hyper-Personalization, you will work on developing and implementing on use cases related to collections with resolution optimization & machine learning models that power advanced personalization capabilities. You will leverage generative AI techniques to create dynamic, tailored user experiences across multiple touchpoints, while optimizing data-driven strategies to enhance customer-collector/channel engagement, resolution rates, and Low operational cost. Key Responsibilities: Generative AI Model Development: Design, develop, and deploy generative AI models use case (e.g., GPT, GANs, VAEs) to enhance personalization in recommen...

We are seeking an experienced Data Engineer (AI/ML Ops) to design, build, and optimize data pipelines, orchestration frameworks, and data infrastructure supporting large-scale AI/ML programs. This role is pivotal in ensuring seamless data flow, from acquisition and preparation to annotation, validation, and delivery enabling AI/ML models to train on clean, well-structured, and compliant datasets at scale. This role will collaborate with Data Scientists, AI Engineers, IT Operations and Product Teams, working across cloud environments (i.e., AWS, GCP, Azure) and integrating with internal and third-party tools to deliver end-to-end data solutions. The ideal candidate brings a blend of technical...

Job Title:Senior Data Engineer Location: Pune, India Job Type: Full-Time As a Senior Data Engineer at Assent, you will play a critical role in advancing our data engineering practices and serving as a hands-on technical expert. You will contribute significantly to the vision and execution of our data engineering efforts, ensuring the development of secure, robust, scalable, and high-performing data platforms. Your role will involve driving key technical initiatives, collaborating on architectural decisions, and executing complex data engineering tasks to align our data infrastructure with Assent's business goals. You will actively participate in the design, development, and implementation of...

Responsibilities: * Design, develop & maintain data pipelines using Snowflake, Python, SQL & AWS. * Optimize performance & ensure data accuracy through ETL processes.

You are applying for the position of Senior GCP Cloud Administrator for Forbes Advisor, a part of the Forbes Marketplace initiative. As a Senior GCP Cloud Administrator, your role will involve managing and optimizing various aspects of Google Cloud Platform (GCP) to ensure efficient data management and operational practices. **Responsibilities:** - Manage and configure roles/permissions in GCP IAM by following the principle of least privileged access - Optimize Big Query service through slot assignments, SQL Queries, FinOps practices, troubleshooting critical data queries, etc. - Collaborate with various teams for efficient data management and operational practices in GCP - Create automation...

8+ years of Data Engineering experience in a Production cloud infrastructure AWS services - Glue, Redshift, RDS, CloudWatch, CloudFormation, S3, IAM etc. 5+ years experience working in a Big Data ecosystem processing data; includes file systems, data structures/databases, automation, security, messaging, movement, etc. • 6+years in Glue development and 4+ years of experience in development in Python. • Experience in translation of Data mapping documents into PySpark ETL code compatible to be able to run on Glue. • Proficiency in Python and SQL is a must.

Experience: 7+ Years Location: Bangalore Key Responsibilities Design, develop, and maintain scalable ETL pipelines using AWS Glue and PySpark . Lead the data engineering team , ensuring best practices in coding, testing, and deployment. Implement workflow orchestration solutions for complex data pipelines. Optimize data pipelines and queries for advanced SQL performance. Collaborate with data scientists, analysts, and stakeholders to deliver reliable data solutions. Ensure data quality, governance, and security across systems. Drive continuous improvements in ETL processes and architecture. Requirements 7+ years of hands-on experience in Data Engineering . Strong expertise in AWS Glue , ETL ...

Requirements and responsibilities We are seeking a Senior ETL Data Engineer to join our data engineering team supporting compliance and risk initiatives. This is a mission-critical role focused on building scalable, high-performance data pipelines that deliver trusted, high-quality data to business and regulatory stakeholders. What youll do Design, build, and maintain robust, scalable ETL processes for data acquisition, transformation, and delivery; Collaborate with Data Analysts, Architects, and Compliance stakeholders to translate requirements into efficient ETL solutions; Ensure data quality, integrity, lineage, and traceability across large-scale datasets; Optimize ETL workflows for perf...

Key Responsibilities: • Architect, develop, and scale advanced Conversational AI Systems powered by LLMs , RAG , and Agentic AI workflows • Implement Retrieval-Augmented Generation (RAG) pipelines using tools like LangChain , LangGraph , LlamaIndex , and custom retrieval stacks • Engineer agentic behaviors with tools like AutoGen , CrewAI , LangGraph , or custom frameworks for multi-step reasoning and tool usage • Apply prompt engineering , fine-tuning , and LoRA/QLoRA techniques to customize LLMs (e.g., OpenAI GPT-4/5, Mistral, Claude, LLaMA) • Design pipelines to minimize hallucination and toxic output , ensuring factual accuracy, safety, and alignment • Integrate structured (e.g., MongoDB...