1569 Data Pipeline Jobs - Page 14

Role Overview: HCLTech is seeking a highly talented and self-motivated Enterprise Architect (AI/GenAI/Cloud) to contribute to advancing the technological world through innovation and creativity. As an Enterprise Architect, you will be responsible for designing scalable architecture integrating AI capabilities, building and managing data pipelines, and deploying AI/ML models on cloud platforms. You will also be expected to have a deep understanding of GenAI concepts, including Generative Models, NLP, Image generation, and Reinforcement Learning. Additionally, a strong knowledge of Deep Learning, ML Frameworks, Data Engineering, and cloud platforms (AWS, Azure, Google Cloud) is essential for t...

Role Proficiency: This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Outcomes: Act creatively ...

Overview Annalect is currently seeking a data engineering lead to join our technology team. In this role you will be building data pipelines and developing data set processes. We are looking for people who have a shared passion for technology, design & development, data, and fusing these disciplines together to build cool things. In this role, you will lead teams working on one or more software and data products in the Annalect Engineering Team. You will participate in technical architecture, design and development of software products as well as research and evaluation of new technical solutions. You will also help us drive the vision forward for data engineering projects by helping us exte...

Dear Candidate, We are looking for ETL Developer for Bengaluru location. Job Location: Bengaluru Experience: 3-9 Years Interview Mode: Face to face Interview location : Bengaluru ETL Developer Role Primary Skills : AWS Glue Job , ETL , Python, SQL Skill Set : AWS, AWS Glue Job , ETL , Python , Apache Spark , PySpark, SQL ,Data Pipelines Key Responsibilities: - Design, develop, and maintain robust ETL pipelines using AWS Glue Job, Python, Apache Spark, and SQL for efficient data extraction, transformation, and loading . - Implement scalable solutions for data storage, ensuring optimal performance and reliability on AWS cloud infrastructure . - Collaborate with stakeholders to understand data ...

Technical Skills Hands-on experience in Core Alteryx, Power BI, SQL Created data pipeline for data inflow and outflow Hand on experience with Power BI dashboard, modelling, DAX, Power Automate Hands on experience with Tableau Understanding python and R within Alteryx is plus Process Specific Skills Know-how of BFSI, preferably Insurance Delivery experience in a business aligned IT scenarios Soft skills (Desired) Excellent Presentation, verbal and written communication skills Excellent analytical skills and aptitude for problem solving, including data analysis and validation Able to work independently and as part of a team Soft Skills (Minimum) Good verbal and written communication skills wit...

Hiring a Python Data Engineer to build and maintain on-premise data pipelines using Dagster. Role includes ETL development, SQL/Python work, CI/CD on Kubernetes, edge data processing, and ensuring data quality, governance, and platform reliability

Job Title: Data Engineer Experience: 5+ Years Location: Chennai/Hyderabad/Mumbai/Pune/Bangalore (Hybrid) Key Responsibilities: Python Proficiency: Demonstrate a strong command of Python programming language, actively contributing to the development and maintenance of data engineering solutions. Data Engineering Expertise: Set up and maintain efficient data pipelines, ensuring smooth data flow and integration across systems. Experience with SQL and Data Warehouse/ Data Lakes is required. Contribute to the establishment and maintenance of Data Lakes, implementing industry best practices. Execute data scrubbing techniques and implement data validation processes to ensure data integrity and qual...

As a Python Developer at our company, you will be responsible for developing cutting-edge AI/ML solutions using Python and various frameworks. Here is a detailed breakdown of the key aspects of the role: **Role Overview:** You will be a crucial part of our team, leveraging your expertise in Python development and AI/ML frameworks to drive innovation in our projects. Your primary focus will be on developing AI/ML solutions using frameworks like FastAPI, Flask, or Django, and working with AI/ML frameworks such as PyTorch and TensorFlow. **Key Responsibilities:** - Develop AI/ML solutions using Python frameworks like FastAPI, Flask, or Django. - Work with AI/ML frameworks such as PyTorch and Te...

Responsibilities Build SQL, Vector, Graph, Document, and NoSQL data architectures that directly support GenAI products. Translate complex business and AI requirements into robust, scalable database designs. Work on GenAI pipelines, retrieval-augmented systems, and large-scale structured data platforms. Collaborate in a fast-paced, startup-style environment supported by experienced leadership. Your Impact Design and develop scalable, secure RDBMS solutions (preferably SQL Server). Build advanced SQL components: stored procedures, views, triggers, dynamic SQL, and data pipelines. Conduct data analysis, profiling, and modeling to support GenAI and analytics initiatives. Optimize performance thr...

Role: Data Engineer (Senior/Associate) Experience: 49 Years Location: Mumbai, Bangalore, Trivandrum (Hybrid) Responsibilities Design and maintain scalable data pipelines, models, and ETL workflows. Collaborate with business teams to translate requirements into data-driven solutions. Build and optimize data warehouses, data lakes, and BI environments. Ensure data quality, governance, and performance across systems. Lead design, integration, and deployment of data flows using GCP tools (BigQuery, Dataflow, Dataproc, Composer). Support and enhance reporting, analytics, and dashboard initiatives. Provide production support and mentor team members. Requirements Bachelor’s in Computer Science or r...

Hello Candidate, Greetings from Hungry Bird IT Consulting Services Pvt. Ltd.! We're hiring a Senior data engineer for our client. Job Title: Senior Data engineer Location: Remote Experience: 10-12yrs Job Summary: We are looking for a highly skilled Senior Data Engineer who will design, build, and optimize data pipelines and analytics infrastructure to support our growing data ecosystem. This role requires a deep understanding of data warehousing, cloud technologies, and modern data engineering practices. Key Responsibilities: Collaborate with engineers and technical managers as part of an autonomous, cross-functional team. Build and maintain data pipelines from core systems into the analytic...

Role & responsibilities: Data Pipeline Development: Design, develop, and maintain efficient and scalable data pipelines using Apache Spark (primarily with Java) or Apache Beam or Kubeflow to ingest, process, and transform large datasets from various sources. GCP Infrastructure Management: Build, configure, and manage data infrastructure components on GCP, including BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Cloud Functions. API Development and Maintenance: Develop and maintain RESTful APIs using Spring Boot to provide secure and reliable access to processed data and data services. Data Modeling and Design: Design and implement optimized data models for analytical and operation...

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-...

We are looking for an experienced and highly skilled Senior Data Engineer to lead the design and development of our data infrastructure and pipelines. As a key member of the Data & Analytics team, you will play a pivotal role in scaling our data ecosystem, driving data engineering best practices, and mentoring junior engineers. This role is ideal for someone who thrives on solving complex data challenges and building systems that power business intelligence, analytics, and advanced data products. Key Responsibilities Design and build robust, scalable, and secure data pipelines and models. Lead the complete lifecycle of ETL/ELT processes, encompassing data intake, transformation, and storage....

We are seeking an experienced and highly skilled Senior Data Engineer to join our organization. The successful candidate will demonstrate deep technical expertise in Snowflake, strong analytical and problem-solving abilities, and the capability to manage client interactions with professionalism and clarity. This role requires a proactive individual who can deliver robust data solutions, collaborate effectively across teams, and adapt to evolving technologies. Design, develop, and maintain efficient, scalable data pipelines and solutions using Snowflake. Collaborate with clients and internal stakeholders to understand business requirements and translate them into effective technical solutions...

Description: We are seeking an technical leader for a hybrid Senior Programmer and Data engineer & Solutions Architect role. This position is for a hands-on expert who not only possesses mastery in Apache Spark development but also has a proven track record of architecting complex, end-to-end data solutions on AWS and on-premise. You will be the technical leader for new data initiatives, responsible for designing the architectural blueprints for our data lake platforms and then leading the hands-on implementation of those designs. We need a strategic thinker who can solve our most complex technical challenges and proactively drive things to add values to the domain.. Requirements: Core Techn...

Description : We are a technology consulting firm operating in Cloud Data Engineering and Analytics, helping enterprise customers build reliable, scalable data platforms and analytics products. Our teams deliver end-to-end data lakes, real-time streaming pipelines, and production-grade ML feature stores using Databricks and modern cloud data tooling. Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage ...

Role : Data Scientist AIML Experience : 4 to 13 Years Relevant Experience : 2+ Years Location : Pan India Notice Period : Immediate to 60 days Mode of Interview - In-Person (29th November 2025) Job Description: 1. Be a hands on problem solver with consultative approach, who can apply Machine Learning & Deep Learning algorithms to solve business challenges Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem Improve Model accuracy to deliver greater business impact Estimate business impact due to deployment of model 2. Work with the domain/customer teams to understand business context , data dictionaries ...

Roles and Responsibilities Design, develop, test, deploy, and maintain large-scale data pipelines using DataBricks, Spark, SQL, and Python. Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs. Develop scalable APIs using Django or Flask for data ingestion and processing. Ensure high-quality code by writing unit tests and integrating with CI/CD pipelines. Troubleshoot issues related to data pipeline failures and optimize performance.

Company Name: TopMNC Position Name: ETL Testing/QA Engineer Experience: 4 to 8 yrs & 8 to 12 yrs Location: Chennai & Bangalore Primary Skills: Hands on Exp in ETL Testing with BigData, Spark & Python Job Description Strong hands-on experience in ETL Testing and data validation. Expertise in Big Data testing using: Spark / PySpark Hive (HiveQL) HDFS Strong SQL and query-writing skills (complex joins, aggregations, window functions). Experience with Python for automation, scripting, or data validation workflows. Understanding of ETL tools (Informatica, Talend, DataStage, ADF, Glue, or equivalent). Familiarity with ETL/Data pipeline concepts (source-to-target mapping, data transformations, line...

Role & responsibilities: Design, develop, and maintain scalable AI-driven applications and services. Collaborate with data scientists and product teams to integrate machine learning models into production systems. Build intuitive front-end interfaces using React to support AI workflows and data visualization. Analyze and process large datasets to support model training and evaluation. Rapidly learn and apply new AI frameworks, tools, and methodologies Preferred candidate profile: Strong proficiency in Python, with experience in data manipulation libraries (e.g., Pandas, NumPy) and backend frameworks (e.g., FastAPI, Flask). Hands-on experience with data engineering and analytics, including da...

We are a technology consulting firm operating in Cloud Data Engineering and Analytics, helping enterprise customers build reliable, scalable data platforms and analytics products.Our teams deliver end-to-end data lakes, real-time streaming pipelines, and production-grade ML feature stores using Databricks and modern cloud data tooling. Role & Responsibilities : - Design, build, and maintain scalable batch and streaming ETL pipelines on Databricks using Delta Lake and Delta Live Tables (DLT). - Develop and optimize Spark/PySpark jobs for performance, cost-efficiency, and reliability; tune cluster sizing and autoscaling policies. - Implement data quality, observability, lineage and monitoring ...

About The Opportunity : A fast-scaling technology company operating in the Enterprise AI / Generative AI sector, building production-grade LLM-driven products and intelligent automation for global clients. We deliver secure, low-latency generative systems that power conversational AI, summarization, code generation, and retrieval-augmented applications across cloud-native environments. We are hiring a Senior Generative AI Engineer (7+ years) to own architecture, model development, and production deployment of advanced generative systems. This is a fully remote role for candidates based in India. Role & Responsibilities : - Lead end-to-end design and delivery of Generative AI/LLM solutionsdat...

About The Opportunity : A fast-scaling technology company operating in the Enterprise AI / Generative AI sector, building production-grade LLM-driven products and intelligent automation for global clients. We deliver secure, low-latency generative systems that power conversational AI, summarization, code generation, and retrieval-augmented applications across cloud-native environments. We are hiring a Senior Generative AI Engineer (7+ years) to own architecture, model development, and production deployment of advanced generative systems. This is a fully remote role for candidates based in India. Role & Responsibilities : - Lead end-to-end design and delivery of Generative AI/LLM solutionsdat...

About The Opportunity : A fast-scaling technology company operating in the Enterprise AI / Generative AI sector, building production-grade LLM-driven products and intelligent automation for global clients. We deliver secure, low-latency generative systems that power conversational AI, summarization, code generation, and retrieval-augmented applications across cloud-native environments. We are hiring a Senior Generative AI Engineer (7+ years) to own architecture, model development, and production deployment of advanced generative systems. This is a fully remote role for candidates based in India. Role & Responsibilities : - Lead end-to-end design and delivery of Generative AI/LLM solutionsdat...