874 Data Pipeline Jobs - Page 35

Key Responsibilities : - Design and develop scalable PySpark pipelines to ingest, parse, and process XML datasets with extreme hierarchical complexity. - Implement efficient XPath expressions, recursive parsing techniques, and custom schema definitions to extract data from nested XML structures. - Optimize Spark jobs through partitioning, caching, and parallel processing to handle terabytes of XML data efficiently. - Transform raw hierarchical XML data into structured DataFrames for analytics, machine learning, and reporting use cases. - Collaborate with data architects and analysts to define data models for nested XML schemas. - Troubleshoot performance bottlenecks and ensure reliability in...

Position : Data Engineer Senior Consultant Location : Hyderabad (Work From Office) Shift : Night Shifts(08:30 PM to 05:30 AM) Two way cab provided Overall 12+ years of experience with at least 5+ years of experience in working with SQL on MySQL and PostgreSQL databases. Must be skilled in working with advanced SQL functions such as CTEs, window functions, procedural SQL, stored procedures, triggers etc. Significant experience in setting up, securing, and administering databases on premise or cloud (MS Azure). Demonstrated experience in managing data and related infrastructure on MS Azure. Demonstrated experience in building and orchestrating performant, secure, and cost-effective data pipeli...

Key Responsibilities Design, develop, and maintain scalable ETL pipelines using Azure Data Factory , Azure Databricks , and other Azure services. Implement data ingestion, transformation, and integration workflows from multiple structured/unstructured sources. Work with Azure Data Lake Storage , Azure SQL Database , and Azure Synapse Analytics for data warehousing and analytics. Perform data modeling (star/snowflake schemas), database management, and query optimization. Ensure data security and compliance within Azure environments. Collaborate with cross-functional teams to align data solutions with business needs. Participate in DevOps practices for CI/CD deployment of data engineering solu...

We are looking for an experienced and visionary Senior Data Scientist / Lead to join our data science team and drive innovation in the banking and fintech space. This role demands a strategic thinker with a strong technical foundation, leadership skills, and deep domain expertise to build scalable AI/ML solutions that influence core business decisions. Key Responsibilities: Design, develop, and implement machine learning models and statistical algorithms to address complex business challenges in banking and fintech. Analyze large and diverse datasets to derive actionable insights and enable data-driven decision-making across functions. Lead, mentor, and guide a team of junior data scientists...

Data Consultant: Ability to clearly identify and communicate potential data and analytics problems with business stakeholders and end users Translate business problems into analytical solutions leveraging Data management / Business Intelligence / Analytics techniques Anticipate, identify, structure, and suggest functional solutions for data and analytical problems Identify and provide suggestions in extracting, integrating, and analyzing data Analyze large datasets using SQL and/or Visualization tools Conduct quantitative analysis and interpret results Identify and design key performance metrics and indicators Data Profiling in data lake, files or in any sources. Build data mapping documents...

Skills Required: Full stack + Data Engineer Position Spring Boot, Angular, Cloud Computing Skills Preferred: Google Cloud Platform , Data Flow, Dataproc, Data Fusion, , Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API Experience Required: 5+ years of overall experience with proficiency in Java, angular or any javascript technology with experience in designing and deploying cloud-based data pipelines and microservices using GCP tools like , Dataflow, and Dataproc. Ability to leverage best in-class data platform technologies (Apache Beam, Kafka, ) to deliver platform features, and design & orchestrate platform services to deliver data platform capabilities. Service-Oriented Ar...

Job Title: Senior Principal Data Engineer Location: Gurgaon, Bangalore Work Schedule: 12:00 PM 8:30 PM IST Job Type: Full-Time Position Overview: We are seeking a Senior Principal Data Engineer with expertise in cloud-native AI/ML architecture and Generative AI . This role will be pivotal in designing innovative solutions, guiding development teams, and ensuring alignment with architectural best practices while leveraging cutting-edge technologies and AWS cloud services. Key Responsibilities: Architect Generative AI solutions using AWS services: Bedrock , SageMaker , Kendra , S3 , PGVector. Lead end-to-end solution design for AI/ML initiatives and complex data systems. Collaborate with cross...

Requirements: Experience : 7+ years of experience in data governance, data management, or related fields. Proven track record in implementing data governance frameworks and policies at an enterprise level. In-depth knowledge of data governance concepts, including data quality, data lineage, metadata management, and data security. Technical Skills : Experience with data governance tools such as Collibra, Informatica, Alation, or similar. Strong understanding of databases, data warehousing, and big data platforms (e.g., Hadoop, Spark). Familiarity with data integration, ETL processes, and data modeling. Proficiency in SQL and other scripting languages (e.g., Python, Shell). Regulatory Knowledg...

Role - Data Engineer Experience - 5+ Years Location - Chennai ( Hybrid) DLF office Shift timing - 6 PM - 3 AM Must-Have Skills: Strong Python programming skills Very strong SQL skills Experience in ETL/ELT development Knowledge of relational databases Ability to work independently and communicate effectively Good-to-Have: Knowledge of data quality frameworks and testing strategies Experience with version control systems like Git Familiarity with CI/CD pipelines for data engineering

Job Title: Data Engineer Tester (Azure & Databricks) Locations: Hyderabad, Pune, Bangalore Must-Have Skills: Azure Data Factory, Databricks Experience: 4+ years in data engineering or QA, 2+ years in data pipeline testing Key Responsibilities: Test ETL/ELT pipelines using Azure Data Factory and Databricks. Validate data transformations across Medallion architecture (Bronze, Silver, Gold). Automate data quality checks using PySpark, SQL, Great Expectations, or pytest. Integrate tests into CI/CD pipelines (Azure DevOps, GitHub Actions). Monitor data quality and pipeline health using Azure tools. Collaborate with data engineers and QA teams; maintain test documentation. Required Skills: Hands-o...

Job Overview : The Data Engineer will be responsible for building and maintaining scalable and robust data pipelines that transform raw data into usable formats for analytics and business decision-making. Key Responsibilities : - Data Pipeline Development : Design, construct, and maintain scalable ETL pipelines using tools like Azure Data Factory. - Stakeholder Collaboration : Work closely with technical teams and business stakeholders to understand data requirements. - Data Analysis & Modeling : Analyze data patterns, apply data modeling best practices (including Star Schema), and ensure quality metadata management. - Delivery & Innovation : Follow industry best practices in delivery and id...

Overview Our team is responsible for building and maintaining the Fund ESG Ratings platform, a business-critical component that powers MSCI’s ESG offerings at the fund level. We manage the end-to-end lifecycle of fund-level ESG data—from ingestion, transformation, and scoring, to final delivery to internal consumers and external clients. Key responsibilities of the team include: Designing scalable data pipelines using Databricks, PySpark, and BigQuery. Supporting regulatory-compliant ESG scoring and reporting logic. Partnering with ESG Research and Index teams to translate methodology changes into platform features. Ensuring data quality, platform stability, and SLA adherence. Driving automa...

Key Responsibilities: Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient ETL/ELT data pipelines using Scala and Apache Spark for large-scale batch and real-time data processing. Real-time Streaming: Develop and manage high-throughput, low-latency data ingestion and streaming applications using Apache Kafka (producers, consumers, Kafka Streams, or ksqlDB where applicable). Spark Expertise: Apply in-depth knowledge of Spark internals, Spark SQL, DataFrames API, and RDDs. Optimize Spark jobs for performance, efficiency, and resource utilization through meticulous tuning (e.g., partitioning, caching, shuffle optimizations). Data Modeling & SQL: Design and impl...

Must have : - Strong on programming languages like Python, Java - One cloud hands-on experience (GCP preferred) - Experience working with Dockers - Environments managing (e.g venv, pip, poetry, etc.) - Experience with orchestrators like Vertex AI pipelines, Airflow, etc. - Understanding of full ML Cycle end-to-end - Data engineering, Feature Engineering techniques - Experience with ML modelling and evaluation metrics - Experience with Tensorflow, Pytorch or another framework - Experience with Models monitoring - Advance SQL knowledge - Aware of Streaming concepts like Windowing , Late arrival , Triggers etc. Good to have : - Hyperparameter tuning experience. - Proficient in either Apache Spa...

Job Summary: We are seeking a highly skilled Senior Data Engineer with 8-10 years of experience in building scalable data solutions and a solid understanding of AI/ML workflows. The ideal candidate will have expertise in data engineering , cloud platforms , and machine learning integration , and will work closely with data scientists, ML engineers, and software developers to operationalize intelligent data systems. Key Responsibilities: Design and implement data pipelines for structured, semi-structured and unstructured data from diverse sources (real-time and batch). Develop and maintain data lakes, data warehouses, and feature stores for scalable ML model training and inference. Collaborat...

Hi Team, We are Looking for a Candidate with Python and AWS Experience Only those candidates revert who is Having at least 4+ Years of Exp in Python and AWS Kanishk.mittal@thehrsolution.in Mail me your Updated CV here. Job Summary: As a Python Developer with AWS , you will be responsible for developing cloud-based applications, building data pipelines, and integrating with various AWS services. You will work closely with DevOps, Data Engineering, and Product teams to design and deploy solutions that are scalable, resilient, and efficient in an AWS cloud environment. Notice Period Immediate to 45 days Key Responsibilities: Python Development : Design, develop, and maintain applications and se...

Position Overview: We are seeking a highly skilled and experienced Software Engineer with 4- 8 years of professional experience in Python and Django, specifically in building REST APIs using frameworks like FASTAPI and Django Rest Framework (DRF). The ideal candidate should have hands-on experience with Redis cache, Docker, PostgreSQL, Kafka, Elasticsearch, and ETL pipelines. Key Responsibilities: Collaborate with cross-functional teams to design, develop, and maintain high-quality software solutions using Python, Django (including DRF), FastAPI, and other modern frameworks. Build robust and scalable REST APIs, ensuring efficient data transfer and seamless integration with frontend and third...

Key responsibilities include: Data Architecture: Perform all technical aspects of data architecture and database management for ID&A, including developing data pipelines, new database structures and APIs as applicable Data Design: Translate logical data architectures into physical data designs, ensuring alignment with data modeling best practices and standards Data Process and Monitoring: Ensure proper data ingestion, validation, testing, and monitoring for ID&A data lake Support database and data platform administration for initiatives building, enhancing, or maintaining databases, data warehouses and data pipelines Data Migration: Design and support migration to a data model that is indepe...

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. This opportunity is in American Express Data Platform and Capabilities team, responsible for supporting new products and services which develop new business opport...

Description: Job Description: We are looking for an experienced Python Data Engineer with a strong background in Python, Pandas, AWS, and SQL to join our data engineering team. The ideal candidate should be passionate about building robust data pipelines, optimizing data workflows, and working on scalable cloud-based data solutions. Requirements: Required Skills: 5.5 to 8 years of hands-on experience in Python programming with strong expertise in Pandas for data manipulation. Solid understanding and experience with AWS cloud services (such as S3, Lambda, Glue, Redshift, Athena, etc.). Proficient in SQL with experience in querying and optimizing large datasets. Good understanding of data ware...

As a Senior Data Engineer at IndiaMart, you will be responsible for designing, implementing, and optimizing data pipelines, ETL processes, and data warehousing solutions. You will collaborate closely with cross-functional teams including product managers, data scientists, analysts, and software engineers to ensure the availability, accuracy, and efficiency of our data ecosystem. The role requires a deep understanding of data engineering best practices, modern data technologies, and a passion for building scalable and robust data systems. Responsibilities: Design, develop, and maintain efficient and scalable data pipelines to support various data initiatives, ensuring the timely delivery of h...

Overview Annalect is currently seeking a data engineering lead to join our technology team. In this role you will be building data pipelines and developing data set processes. We are looking for people who have a shared passion for technology, design & development, data, and fusing these disciplines together to build cool things. In this role, you will lead teams working on one or more software and data products in the Annalect Engineering Team. You will participate in technical architecture, design and development of software products as well as research and evaluation of new technical solutions. You will also help us drive the vision forward for data engineering projects by helping us exte...

**Key Responsibilities:** - Design, develop, and maintain scalable data pipelines and workflows using Azure Pipelines. - Build and optimize data platforms to support analytics and business intelligence initiatives. - Collaborate with stakeholders to understand data requirements and translate them into technical solutions. - Implement best practices for data engineering, including data modeling, ETL processes, and performance optimization. - Ensure data security, integrity, and compliance with industry standards. - Monitor and troubleshoot data systems to ensure reliability and efficiency. - Stay updated on emerging technologies and trends in data engineering to drive innovation. **Requiremen...

Jidoka Technologies is looking for Data Science Engineer to join our dynamic team and embark on a rewarding career journey Data Exploration and Preparation:Explore and analyze large datasets to understand patterns and trends Prepare and clean datasets for analysis and model development Feature Engineering:Engineer features from raw data to enhance the performance of machine learning models Collaborate with data scientists to identify relevant features for model training Model Development:Design and implement machine learning models to solve business problems Work on both traditional statistical models and modern machine learning algorithms Scalable Data Pipelines:Develop scalable and efficie...