1166 Data Pipeline Jobs - Page 8

Role & responsibilities Must have experience in Data Lake , Data Bricks and Pyspark Preferred candidate profile

Job Title : Data & AI-backed QA Engineer We are seeking a highly skilled Automation, Data & AI-backed QA Engineer with proven expertise in automation testing, data pipeline validation, and AI-driven quality assurance practices. The ideal candidate will bring deep knowledge of test automation frameworks, strong data testing expertise, and innovative approaches leveraging AI/ML to enhance quality, coverage, and defect prediction. Key Responsibilities Design, develop, and maintain automation test frameworks for web, API, and data- centric applications. Perform data pipeline testing (ETL/ELT validation, schema verification , data integrity, reconciliation, and transformations across multiple sou...

Software Engineer (3–6 yrs), Scala Programming, Apache Spark, SQL, Backend, Functional Programming, Performance Tuning. Exp in Tableau/Zeppelin is preferred. C2H via TE Infotech (Nutanix), Convertible 2 Permanent, BLR. Apply: ssankala@toppersedge.com

Overview of 66degrees 66degrees is a leading consulting and professional services company specializing in developing AI-focused, data-led solutions leveraging the latest advancements in cloud technology. With our unmatched engineering capabilities and vast industry experience, we help the world's leading brands transform their business challenges into opportunities and shape the future of work. At 66degrees, we believe in embracing the challenge and winning together. These values not only guide us in achieving our goals as a company but also for our people. We are dedicated to creating a significant impact for our employees by fostering a culture that sparks innovation and supports professio...

Serko is a cutting-edge tech platform in global business travel & expense technology. When you join Serko, you become part of a team of passionate travellers and technologists bringing people together, using the world's leading business travel marketplace. We are proud to be an equal opportunity employer, we embrace the richness of diversity, showing up authentically to create a positive impact. There's an exciting road ahead of us, where travel needs real, impactful change. With offices in New Zealand, Australia, North America, and China, we are thrilled to be expanding our global footprint, landing our new hub in Bengaluru, India. With a rapid growth plan in place for India, we're hiring p...

Serko is a cutting-edge tech platform in global business travel & expense technology. When you join Serko, you become part of a team of passionate travellers and technologists bringing people together, using the world's leading business travel marketplace. We are proud to be an equal opportunity employer, we embrace the richness of diversity, showing up authentically to create a positive impact. There's an exciting road ahead of us, where travel needs real, impactful change. With offices in New Zealand, Australia, North America, and China, we are thrilled to be expanding our global footprint, landing our new hub in Bengaluru, India. With a rapid growth plan in place for India, we're hiring p...

Role & responsibilities Architecture/ Design Decisions & Technology Evaluations - Industry Trends, Research, POC, Tools - Test Plan Reviews - Production Support Transition Reviews - Contribute to our architecture community by attending and participating in Architecture & Design Roundtables, Architecture Review Boards, and Solution Architecture Guild meetings. The Minimum Qualifications Bachelors or Masters degree in Computer Science, Information Systems, or related field. 8+ years of experience in data architecture and engineering, with a strong focus on financial services. Proven experience with lake house platforms and cloud-native data ecosystems Strong understanding of wealth management ...

Purpose As a Senior Data Engineer at LogixHealth, you will work with a globally distributed team of engineers to design and build cutting edge solutions that directly improve the healthcare industry. Youll contribute to our fast-paced, collaborative environment and bring your expertise to continue delivering innovative technology solutions, while mentoring others. Duties and Responsibilities 1. Lead and contribute to the creation of a self-service data platform for reporting and analytics 2. Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake in the Azure ecosystem (Blob Storage, Data Factory, Event Hubs) 3. Ensure best practices for ETL / ELT processes (data...

-Design, develop & maintain data pipelines using GCP services: Dataflow, BigQuery, and Pub/Sub -Provisioning infrastructure on GCP using IaC with Terraform -Implement & manage data warehouse solutions -Monitor and resolve issues in data workflows Required Candidate profile -Expertise in GCP, Apache Beam, Dataflow, & BigQuery -Pro in Python, SQL, PySpark -Worked with Cloud Composer for orchestration -Solid understanding of DWH, ETL pipelines, and real-time data streaming

-Design, develop & maintain data pipelines using GCP services: Dataflow, BigQuery, and Pub/Sub -Provisioning infrastructure on GCP using IaC with Terraform -Implement & manage data warehouse solutions -Monitor and resolve issues in data workflows Required Candidate profile -Expertise in GCP, Apache Beam, Dataflow, & BigQuery -Pro in Python, SQL, PySpark -Worked with Cloud Composer for orchestration -Solid understanding of DWH, ETL pipelines, and real-time data streaming

-Design, develop & maintain data pipelines using GCP services: Dataflow, BigQuery, and Pub/Sub -Provisioning infrastructure on GCP using IaC with Terraform -Implement & manage data warehouse solutions -Monitor and resolve issues in data workflows Required Candidate profile -Expertise in GCP, Apache Beam, Dataflow, & BigQuery -Pro in Python, SQL, PySpark -Worked with Cloud Composer for orchestration -Solid understanding of DWH, ETL pipelines, and real-time data streaming

Skill: Pyspark Experience: 6 to 13 years Location: Bhubaneswar Job description : Develop and maintain scalable data pipelines using Python and PySpark. Collaborate with data engineers and data scientists to understand and fulfill data processing needs. Optimize and troubleshoot existing PySpark applications for performance improvements. Write clean, efficient, and well-documented code following best practices. Participate in design and code reviews. Develop and implement ETL processes to extract, transform, and load data. Ensure data integrity and quality throughout the data lifecycle. Stay current with the latest industry trends and technologies in big data and cloud computing. Qualificatio...

Skill: Pyspark Experience: 6 to 13 years Location: Bhubaneswar Job description : Develop and maintain scalable data pipelines using Python and PySpark. Collaborate with data engineers and data scientists to understand and fulfill data processing needs. Optimize and troubleshoot existing PySpark applications for performance improvements. Write clean, efficient, and well-documented code following best practices. Participate in design and code reviews. Develop and implement ETL processes to extract, transform, and load data. Ensure data integrity and quality throughout the data lifecycle. Stay current with the latest industry trends and technologies in big data and cloud computing. Qualificatio...

Skill: Pyspark Experience: 6 to 9 years Location: Kolkata Job description : Responsibilities Develop and maintain scalable data pipelines using Python and PySpark. Collaborate with data engineers and data scientists to understand and fulfill data processing needs. Optimize and troubleshoot existing PySpark applications for performance improvements. Write clean, efficient, and well-documented code following best practices. Participate in design and code reviews. Develop and implement ETL processes to extract, transform, and load data. Ensure data integrity and quality throughout the data lifecycle. Stay current with the latest industry trends and technologies in big data and cloud computing. ...

As a Product Analyst II, you will play a pivotal role in transforming data into actionable insights to drive product decisions. Partner closely with product managers, engineers, and designers to inform roadmap priorities, improve user experience, and measure product success. This role is ideal for someone with 2-4 years of analytics experience looking to grow into a senior analytics or product domain. - Conduct cohort analysis, funnel analytics, and broader product analytics to deeply understand user behavior, engagement patterns, and drop-off points. - Collaborate with Product Managers to define success metrics for new and existing features. - Perform exploratory deep-dives and structured a...

Experience Required : Minimum 7+ Years (Mandatory) Job Type : Full-time | Day Shift (IST) Job Description : We are looking for a skilled and experienced Databricks Engineer to join our team at our Hyderabad (Madhapur) office. The ideal candidate will have a strong background in data engineering, hands-on experience with Databricks, and a solid understanding of cloud platforms (AWS, Azure, or GCP). This is a great opportunity to work on cutting-edge data projects in a collaborative and fast-paced environment. Key Responsibilities : - Design and implement scalable and efficient data pipelines, ETL/ELT processes, and data integration solutions using Databricks. - Work with large-scale structure...

Position : Data Engineer Senior Consultant Location : Hyderabad (Work From Office) Shift : Night Shifts(08:30 PM to 05:30 AM) Two way cab provided Overall 12+ years of experience with at least 5+ years of experience in working with SQL on MySQL and PostgreSQL databases. Must be skilled in working with advanced SQL functions such as CTEs, window functions, procedural SQL, stored procedures, triggers etc. Significant experience in setting up, securing, and administering databases on premise or cloud (MS Azure). Demonstrated experience in managing data and related infrastructure on MS Azure. Demonstrated experience in building and orchestrating performant, secure, and cost-effective data pipeli...

An exciting opportunity for a skilled and motivated Data Scientist to join a growing group (AXIS, Access eXtract and Integrate Safe Data) within the Clinical Informatics Research Unit (CIRU) This role will continue to establish and grow The Pre-hospital Research and Audit Network (PRANA) PRANA is creating an end to end research ready database Currently there is no method for undertaking research at scale from pre hospital critical care all the way through to patient outcomes Additional connections to other datasets such as traffic collision databases will also be incorporated into PRANA By creating such a resource, the ability to identify life saving trends and treatments in medicine and roa...

Role & responsibilities Understand business processes, a bigger picture and core ideas behind the developed software. Support business analysts and developers with data subject expertise, query building and optimization Educate and evangelize application teams of the best practices on query building and optimization, data modeling and design Develop a thorough understanding of the data definitions, domain values, data relationships, business rules, sources, and data integration for each subject area Advocate and advance toward cutting edge engineering technologies Enhancing data collection procedures to include information that is relevant for building analytic systems Processing, cleansing,...

At Barclays, we dont just adapt to the future we shape it Embark on a transformative journey as Senior AWS Data Engineer Vice President, where you'll spearhead the evolution of our Reference Data Services function You will be part of a team of engineers with accountability and responsibility for the engineering (strategy, architecture, design, build, documentation, and testing) of the software components You will work closely with your colleagues in the wider cloud and middleware organization to bring their products to life as part of a modern developer-focused experience Additionally, your role requires effective stakeholder management, leadership, and decision-making to support business st...

An exciting opportunity for a skilled and motivated Data Scientist to join a growing group (AXIS, Access eXtract and Integrate Safe Data) within the Clinical Informatics Research Unit (CIRU) This role will continue to establish and grow The Pre-hospital Research and Audit Network (PRANA) PRANA is creating an end to end research ready database Currently there is no method for undertaking research at scale from pre hospital critical care all the way through to patient outcomes Additional connections to other datasets such as traffic collision databases will also be incorporated into PRANA By creating such a resource, the ability to identify life saving trends and treatments in medicine and roa...

Role: Senior Snowflake Engineer A Senior Snowflake Developer with Cortex experience is responsible for designing, building, and optimizing data solutions on the Snowflake platform, with a specific focus on leveraging Snowflake Cortex for AI-driven features like RAG (Retrieval- Augmented Generation), LLM functions, and vector search. Key responsibilities include developing scalable ETL/ELT pipelines, creating data models, implementing AI assistants, and coaching other team members, all while adhering to best practices in performance, cost control, and security. Skills: • Bachelors degree in Computer Science, Software Engineering, Information Technology, or related field required. • At least 4...

Salary: 15 to 30 LPA Exp: 3 to 8 years Location : Gurgaon/Bangalore/Pune/Chennai Notice: Immediate to 30 days..!! Key Responsibilities & Skillsets: Common Skillsets : 3+ years of experience in analytics, Scala, Pyspark, Python, Spark, SQL and associated data engineering jobs. Must have experience with managing and transforming big data sets using pyspark, spark-scala, Numpy pandas Excellent communication & presentation skills Experience in managing Python codes and collaborating with customer on model evolution Good knowledge of data base management and Hadoop/Spark, SQL, HIVE, Python (expertise). Superior analytical and problem solving skills Should be able to work on a problem independentl...

Job Title: AWS Data Engineer Company Name: NCSi Job Description: NCSi is seeking a skilled AWS Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing and building robust data pipelines, developing data integration processes, and optimizing database performance in cloud environments. You will work collaboratively with data scientists, analysts, and other stakeholders to ensure data accuracy, reliability, and accessibility across the organization. Your role will also involve implementing best practices in data governance and security, as well as leveraging AWS services to support data analytics and reporting needs. Key Responsibilities: - Design, develop,...

skilled and motivated Data Engineer with hands-on experience in cloud platforms like AWS, Azure, or GCP to join our growing data team. have strong knowledge of data architecture, pipeline development, and cloud-based data processing.