3177 Data Pipelines Jobs - Page 8

We are looking for a highly skilled Senior Data Engineer to join our team The ideal candidate will have hands-on experience working with large-scale data platforms and a strong background in Python, PySpark, AWS, and modern data warehousing tools such as Snowflake and DBT Familiarity with NoSQL databases like MongoDB and real-time streaming platforms like Kafka is essential Key Responsibilities: Design, build, and maintain scalable data pipelines using PySpark and Python Work with AWS cloud services (S3, Lambda, Glue, EMR, Redshift) for data ingestion, processing, and storage Implement and maintain ETL workflows using DBT and orchestration tools (e g , Airflow) Design and manage data models ...

Job Purpose and Impact The Professional, Data Engineering job designs, builds and maintains moderately complex data systems that enable data analysis and reporting. With limited supervision, this job collaborates to ensure that large sets of data are efficiently processed and made accessible for decision making. Key Accountabilities DATA & ANALYTICAL SOLUTIONS: Develops moderately complex data products and solutions using advanced data engineering and cloud based technologies, ensuring they are designed and built to be scalable, sustainable and robust. DATA PIPELINES: Maintains and supports the development of streaming and batch data pipelines that facilitate the seamless ingestion of data f...

We are seeking a highly skilled Python Developer with strong expertise in AWS Athena to join our data engineering team with 5years to 7 years experience The successful candidate will be responsible for building and maintaining data pipelines, optimizing queries on large-scale datasets, and integrating AWS Athena with Python-based applications This role is ideal for developers who thrive in cloud-native, data-intensive environments

Design, develop, and maintain data pipelines and ETL processes using AWS and Snowflake Implement data transformation workflows using DBT (Data Build Tool) Write efficient, reusable, and reliable code in Python Optimize and tune data solutions for performance and scalability Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions Ensure data quality and integrity through rigorous testing and validation Stay updated with the latest industry trends and technologies in data engineering Bachelor''s or Master''s degree in Computer Science, Engineering, or a related field Proven experience as a Data Engineer or similar role Strong pro...

Assist in building and testing AI models for knowledge generation and resolution recommendation Develop scripts and connectors for ITSM tool integration Support data preparation and model training activities Document workflows and assist in creating knowledge artifacts Participate in validation workshops with SMEs Required Skills: Exposure to LLMs and AIdriven automation Basic understanding of ITIL processes and service desk operations Knowledge of Python and API integration Solid experience with LLMs, embeddings, RAG pipelines for IT Ops use cases Strong Python coding skills with ability to build integrations, data pipelines, and automation scripts Good understanding of ServiceNowJira integ...

Position Title: Business Analyst II Department : Product Title of Supervisor: Manager Product Management PURPOSE AND SCOPE: The Business Analyst provides health operations and business systems expertise during the receipt, review and delivery of client requests and internal projects related to data integrations, reporting and analytics for the claims administration solution. PRINCIPAL RESPONSIBILITIES AND DUTIES: Manage resolution of incoming client tickets, including gathering information, recreation of issues, resolution of the issue and co-ordination with developers Analyze incoming projects, client and strategic, requests and gather business requirements Ensure appropriate communication ...

Build and optimize data pipelines and AI modelsPerform statistical and predictive analysisDevelop OCR, NLP, and anomaly detection solutions Apply GEN AI and prompt engineering Visualize insights using Power BI/TableauWork with cloud platforms (AWS, Azure, GCP) Collaborate cross-functionally to improve decisions

Design and develop robust ETL pipelines using Python, PySpark, and GCP services Build and optimize data models and queries in BigQuery for analytics and reporting Ingest, transform, and load structured and semi-structured data from various sources Collaborate with data analysts, scientists, and business teams to understand data requirements Ensure data quality, integrity, and security across cloud-based data platforms Monitor and troubleshoot data workflows and performance issues Automate data validation and transformation processes using scripting and orchestration tools Required Skills & Qualifications: Hands-on experience with Google Cloud Platform (GCP), especially BigQuery Strong progra...

WorkMode :Hybrid Work Timing : 2 PM to 11 PM Primary : AWS Data Engineer - AWS Glue, Redshift, Python/Pyspark,SQL, ETL. Design and build scalable data pipelines using AWS services like AWS Glue, Amazon Redshift, and S3. Develop efficient ETL processes for data extraction, transformation, and loading into data warehouses and lakes. Create and manage applications using Python, SQL, Databricks, and various AWS technologies. Automate repetitive tasks and build reusable frameworks to improve efficiency. Skill Proficiency Level expected AWS Data Engineer - AWS Glue, Amazon Redshift, S3 Expert with 8.5 to 10+ years of hands-on experience ETL Process , SQL, Databricks

Job description Design and execute test plans for ETL processes, ensuring data accuracy, completeness, and integrity. Develop automated test scripts using Python or R for data validation and reconciliation. Perform source-to-target data verification, transformation logic testing, and regression testing. Collaborate with data engineers and analysts to understand business requirements and data flows. Identify data anomalies and work with development teams to resolve issues. Maintain test documentation, including test cases, test results, and defect logs. Participate in performance testing and optimization of data pipelines. Required Skills Qualifications: Strong experience in ETL testing acros...

Job description We are seeking an experienced Generative AI Data Scientist with a strong background in LLM training, prompt engineering, and agentic AI systems. The ideal candidate will combine deep technical expertise with innovative thinking to design, train, and deploy cutting-edge AI solutions that drive business impact You will work closely with cross-functional teams to implement and scale generative AI capabilities across the organization LLM Development Optimization Fine-tune and optimize foundation models (e g, GPT, LLaMA, Claude, Mistral) for specific business and domain applications Build scalable pipelines for model training, evaluation, and deployment Conduct performance tuning,...

We are looking for a highly skilled and experienced Data Engineering Tech Lead to join our team at ERNST YOUNG LLP. The ideal candidate will have a strong background in data engineering and technical leadership, with 6-10 years of experience. Roles and Responsibility Design and implement scalable data pipelines and architectures. Lead the development of data engineering solutions using various technologies. Collaborate with cross-functional teams to identify business requirements and develop technical solutions. Develop and maintain large-scale data systems and applications. Ensure high-quality data processing and analysis. Mentor junior engineers and provide technical guidance. Job Requirem...

Job description Primary skills : Gemini Agent space, AI multi cloud experience, Experience in running AI services, vibe coding(Cursor), Vertex Secondary skills : Terraform, Multi cloud experience (AWS) Excellent programming skills and proficiency in at least one of the major programming scripting languages used in Gen AI orchestration such as Python or PySpark or Java Hands on exposure to integrating with Gemini Pro 1.x using API endpoints. Hands on exposure to using GCP services for storage, serverless-logic, search, transcription, and chat Thorough understanding of Cursor coding(Vibe coding); implementation exposure to LLM agents like LangChain & vector databases Pinecone or Chroma or FAIS...

Job description Key Responsibilities Assist in planning and executing the migration of data workloads from OCI to GCP. Analyze existing OCI services and recommend equivalent or improved GCP solutions to optimize cost and performance. Develop and maintain data pipelines and architectures using GCP services such as BigQuery, Cloud Storage, Dataflow, and Pub/Sub. Implement automation and infrastructure-as-code using tools like Terraform, Deployment Manager, or Cloud Build. Collaborate with architects, tech leads, and DevOps teams to ensure smooth migration and integration. Monitor GCP resource usage and implement cost-saving measures. Ensure data security, governance, and compliance throughout ...

Job description Design, develop, and maintain data pipelines and ETL processes using AWS and Snowflake. Implement data transformation workflows using DBT (Data Build Tool). Write efficient, reusable, and reliable code in Python. Optimize and tune data solutions for performance and scalability. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. Ensure data quality and integrity through rigorous testing and validation. Stay updated with the latest industry trends and technologies in data engineering. Bachelor''s or Master''s degree in Computer Science, Engineering, or a related field. Proven experience as a Data Engineer or...

Qualification About the Role We are seeking a highly motivated and experienced Generative AI Lead to spearhead our efforts in leveraging generative AI technologies to solve complex business problems, enhance productivity, and build innovative products and solutions. You will lead the design, development, and deployment of cutting edge GenAI models and systems that deliver tangible business value. Key Responsibilities Strategy and Leadership Define and drive the companys generative AI roadmap in alignment with business goals Collaborate with senior leadership to identify high impact GenAI use cases across departments Evangelize GenAI within the organization, fostering a culture of innovation ...

Job description We are looking for a Data Engineer with 5 to 7 years of experience, possessing strong skills in Python, AWS, SQL, and experience working with any major database. The ideal candidate should demonstrate a solid understanding of development best practices, including peer code reviews and unit testing. A strong grasp of fundamental data modeling concepts such as 3NF, Star Schema, Snowflake Schema, and data grain is essential. Candidates should also have hands-on experience with data analysis and data profiling tasks. Mandatory skills for this role include proficiency in Python, SQL (using Oracle or PostgreSQL), AWS, and experience in building and managing ETL data pipelines

Job description P2-C3-TSTS AWS Data Engineer Design and build scalable data pipelines using AWS services like AWS Glue, Amazon Redshift, and S3. Develop efficient ETL processes for data extraction, transformation, and loading into data warehouses and lakes. Create and manage applications using Python, SQL, Databricks, and various AWS technologies. Automate repetitive tasks and build reusable frameworks to improve efficiency. AWS Data Engineer Design and build scalable data pipelines using AWS services like AWS Glue, Amazon Redshift, and S3. Develop efficient ETL processes for data extraction, transformation, and loading into data warehouses and lakes. Create and manage applications using Pyt...

Job description Position Overview We are seeking a highly skilled Machine Learning ML Lead Developer to design develop and lead the implementation of advanced ML solutions The ideal candidate will combine deep technical expertise in machine learning AI frameworks and software development with leadership skills to guide a team of data scientists and engineers This role requires close collaboration with product managers business stakeholders and cross functional engineering teams to deliver scalable and impactful AI driven solutions Key Responsibilities Leadership Strategy Lead the design development and deployment of ML models and AI driven applications Define the technical roadmap for ML ini...

Design and execute test plans for ETL processes, ensuring data accuracy, completeness, and integrity Develop automated test scripts using Python or R for data validation and reconciliation Perform source-to-target data verification, transformation logic testing, and regression testing Collaborate with data engineers and analysts to understand business requirements and data flows Identify data anomalies and work with development teams to resolve issues Maintain test documentation, including test cases, test results, and defect logs Participate in performance testing and optimization of data pipelines Required Skills & Qualifications: Strong experience in ETL testing across various data source...

Bachelors degree or equivalent in Computer Engineering Computer Science or a related field 5 years of experience in data software engineering role 3 years of strong hands on experience with Databricks with AWS as a cloud platform which including Spark based processing data pipelines and the platforms tools for analytics and machine learning 2 years of experience with building AWS cloud based data pipelines and AI solutions 2 years of Hands on experience with Python and PySpark The role holder will possess a blend of design skills needed for Agile data development projects Proficiency or demonstratable passion for learning in data engineer techniques and testing methodologies Strong written a...

We are seeking a skilled and proactive Data Integration Engineer with handson experience in modern data platforms and integration tools The ideal candidate will be responsible for designing developing and maintaining robust data pipelines and integrations across various systems using tools like SnapLogic Fivetran Snowflake and DBT Key Responsibilities Design and implement scalable data pipelines using SnapLogic Fivetran and DBT Develop and optimize data models and queries in Snowflake using SQL and Python Integrate data from various sources using REST SOAP APIs with appropriate authentication methodsCollaborate with cross functional teams to understand data requirements and deliver high qual...

Joining: Immediate to 15 days Key Skills: Strong hands-on experience in PySpark Expertise in Big Data technologies Solid understanding of SQL, Data Pipelines, and ETL Interested candidates can share their resume with the following details to navaneetha@suzva.com Current CTC Expected CTC Location

About the Role We are looking for a skilled Python Developer with hands-on experience in building and consuming APIs, designing database schemas, and integrating data pipelines that support visualization in Tableau. The ideal candidate will have a strong foundation in Python development, database design and API integration, with an added advantage of working knowledge of Tableau for data visualization. Key Responsibilities : - API Development & Integration : - Design, develop, and maintain RESTful APIs using Python frameworks (e.g., FastAPI , Flask, or Django). Consume third-party APIs and integrate them with internal systems. Optimize data storage and retrieval mechanisms for performance an...

We are seeking a skilled Big Data Developer with 3+ years of experience to develop, maintain, and optimize large-scale data pipelines using frameworks like Spark, PySpark, and Airflow. The role involves working with SQL, Impala, Hive, and PL/SQL for advanced data transformations and analytics, designing scalable data storage systems, and integrating structured and unstructured data using tools like Sqoop. The ideal candidate will collaborate with cross-functional teams to implement data warehousing strategies and leverage BI tools for insights. Proficiency in Python programming, workflow orchestration with Airflow, and Unix/Linux environments is essential. Location: Remote- Delhi / NCR,Banga...