1586 Data Pipeline Jobs - Page 28

Roles and Responsibilities : - Create and maintain efficient and scalable data models as per business needs - Create and maintain optimal data pipelines against multiple data sources lie SQL, BigData on Azure / AWS cloud; - Assemble and process large, complex data sets to meet both functional and non-functional business requirements; - Analyze and improve existing data models, pipelines, related infrastructure and processes - Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics; - Monitor and improve data quality and data governance policies - Collaborate with stakehold...

Role Proficiency: This role requires proficiency in data pipeline development including coding testing and implementing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc along with coding skills in Python PySpark and SQL. Works independently according to work allocation. Outcomes: Operate with minimal guidance to develop error-free code test applications and document the development process. Understand application features and component designs to develop them in accordance with user stories and requirements. Code debug test document and communicate the stages of product...

Role Proficiency: This role requires proficiency in data pipeline development including coding testing and implementing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc along with coding skills in Python PySpark and SQL. Works independently according to work allocation. Outcomes: Operate with minimal guidance to develop error-free code test applications and document the development process. Understand application features and component designs to develop them in accordance with user stories and requirements. Code debug test document and communicate the stages of product...

Role Proficiency: This role requires proficiency in data pipeline development including coding testing and implementing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc along with coding skills in Python PySpark and SQL. Works independently according to work allocation. Outcomes: Operate with minimal guidance to develop error-free code test applications and document the development process. Understand application features and component designs to develop them in accordance with user stories and requirements. Code debug test document and communicate the stages of product...

Role : Senior Databricks Engineer / Databricks Technical Lead/ Data Architect. Experience : 8-15 years. Primary Roles And Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack. - Ability to provide solutions that are forward-thinking in data engineering and analytics space. - Collaborate with DW/BI leads to understand new ETL pipeline development requirements. - Triage issues to find gaps in existing pipelines and fix the issues. - Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs. - Help joiner team members to resolve issues and technical challenges. - Drive technical discussion with...

Experience : 8-15 years. Primary Roles And Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack. - Ability to provide solutions that are forward-thinking in data engineering and analytics space. - Collaborate with DW/BI leads to understand new ETL pipeline development requirements. - Triage issues to find gaps in existing pipelines and fix the issues. - Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs. - Help joiner team members to resolve issues and technical challenges. - Drive technical discussion with client architect and team members. - Orchestrate the data pipelines in schedul...

Job Title: Senior Data Engineer, Microsoft Fabric + SSIS Specialist Location: Noida, Sector 62 ( Onsite/Hybrid, Full-Time, Long-Term Proje ct) Experience Required: 7 + Years Availability: Immediate Joiners Only Role Overview: We are looking for a highly skilled Senior Data Engineer 7+ years of experience as a Microsoft Fabric Specialist to lead the design, development, and optimization of modern data solutions. The role involves working extensively with Azure Fabric, ADF, Databricks, SSIS, Power BI, and PySpark to build scalable, high-performance data pipelines and analytics systems. Key Responsibilities: Lead the design, development, and implementation of Fabric Pipelines, SSIS Packages, Da...

**Job Title: Lead Data Engineer - Data** **Company: Happiest Minds** **Location:** Bangalore **Job Type:** Full-Time **Experience Required:** 5 - 8 years **About Happiest Minds:** Happiest Minds is a next-generation digital transformation and technology services company that enables businesses to achieve their vision by delivering innovative solutions with a human touch. Our mission is to leverage the latest technologies to help our clients stay ahead in the digital era. **Job Description:** We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team at Happiest Minds. The ideal candidate will have a strong background in data engineering with expertise in data...

About The Role : We are seeking a highly skilled and experienced Senior PySpark/Python Developer to play a critical role in building a robust and reliable system for managing and disseminating customer notifications regarding PG&E's Planned Power Outages (PPOs). This is an exciting opportunity to tackle complex data challenges within a dynamic environment and contribute directly to improving customer communication and experience. As a key member of the team, you will be responsible for developing and implementing data processing pipelines that can ingest, transform, and synthesize information from various backend systems related to PPOs. Your primary goal will be to "eat ambiguity and excret...

Quantitative Developer : A Quantitative Developer, often called a "Quant Dev," is a highly specialized professional who works at the intersection of finance, mathematics, and computer science. This role involves designing, implementing, and optimizing algorithms and software solutions for complex financial problems, such as trading strategies, risk management, and pricing models. A successful Quant Dev combines deep programming expertise with a strong grasp of quantitative finance principles. Key Responsibilities : - Develop, implement, and maintain complex quantitative models and algorithms for high-frequency trading, risk management, or portfolio optimization. - Write high-performance, low...

Welcome to Warner Bros. Discovery the stuff dreams are made of. Who We Are When we say, the stuff dreams are made of, we’re not just referring to the world of wizards, dragons and superheroes, or even to the wonders of Planet Earth. Behind WBD’s vast portfolio of iconic content and beloved brands, are the storytellers bringing our characters to life, the creators bringing them to your living rooms and the dreamers creating what’s next From brilliant creatives, to technology trailblazers, across the globe, WBD offers career defining opportunities, thoughtfully curated benefits, and the tools to explore and grow into your best selves. Here you are supported, here you are celebrated, here you c...

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...

Role: Technical Project Manager Experience: 12 to 16 years Notice Period: Immediate to 30 days ONLY Job Description: Should have 12-15 years of hands-on project management experience and leading projects with in-depth knowledge of SDLC. Should be able to lead multiple projects simultaneously. Should be able to communicate across a diverse set of stakeholders (Business stakeholders, third parties, Suppliers). Excellent customer relationship management & collaboration skills. Ability to challenge Ideas and speak up to raise risks, issues. Experience working with solution architects and in setting up the Project / Product roadmaps involving all the stakeholders. Experience in working with proje...

ABOUT FLUTTER ENTERTAINMENT Flutter Entertainment is the worlds largest sports betting and iGaming operator with 13.9 million Average Monthly Players worldwide and an annual revenue of $14Bn in 2024. We have a portfolio of iconic brands, including Paddy Power, Betfair, FanDuel, PokerStars, Junglee Games and Sportsbet. Flutter Entertainment is listed on both the New York Stock Exchange (NYSE) and the London Stock Exchange (LSE). In 2024, we were recognized in TIMEs 100 Most Influential Companies under the 'Pioneers' categorya testament to our innovation and impact. Our ambition is to transform global gaming and betting to deliver long-term growth and a positive, sustainable future for our sec...

Primary Roles And Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack. - Ability to provide solutions that are forward-thinking in data engineering and analytics space. - Collaborate with DW/BI leads to understand new ETL pipeline development requirements. - Triage issues to find gaps in existing pipelines and fix the issues. - Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs. - Help joiner team members to resolve issues and technical challenges. - Drive technical discussion with client architect and team members. - Orchestrate the data pipelines in scheduler via Airflow. Skills An...

Role & Responsibilities : - Design, develop, and maintain robust AWS-based data pipelines using Python and PySpark. - Implement efficient ETL processes, ensuring data integrity and optimal performance across AWS services (S3, Glue, EMR, Redshift). - Collaborate with cross-functional teams to integrate data engineering solutions within broader business-critical applications. - Troubleshoot and optimize existing data workflows, ensuring high availability, scalability, and security of cloud solutions. - Exercise best practices in coding, version control, and documentation to maintain a high standard of engineering excellence. Skills & Qualifications : Must-Have : 4+ years of hands-on experience...

The Senior Data Engineer will be responsible for designing, building, and managing the data infrastructure and data pipeline processes for the bank. This role involves leading a team of data engineers, working closely with data scientists, analysts, and IT professionals to ensure data is accessible, reliable, and secure. The ideal candidate will have a strong background in data engineering, excellent leadership skills, and a thorough understanding of the banking industrys data requirements. Leadership and Team Management . Lead, mentor, and develop a team of data engineers. Establish best practices for data engineering and ensure team adherence. Coordinate with other IT teams, business units...

Ensure adherence to Security and Compliance policies and procedures Collaborate with various stakeholders and assist in architecting, designing and implement the optimal solutions with best practice and standards Expertise in Ataccama DQ tool. Create robust data quality pipeline using Ataccama DQ and ONE Desktop, ensuring it is ready for analytics and reporting. Be proficient in writing scripting or programming language to handle large volume of data processing. Adopt DevOps methodologies and tools for continuous integration and deployment (CI/CD) and automate to enhance data engineering processes. Experience to work on AWS data lakes to store and process both structured and unstructured dat...

Establish and enforce best practices in data pipeline development, code quality, testing, deployment, and monitoring Ensure data solutions comply with security, privacy, and governance policies Optimize the performance and costefficiency of data processing workflows Drive continuous improvement by evaluating and adopting new technologies and tools within the GCP ecosystem

Build and Maintain Data Architectures: Develop and maintain data pipelines that enable the transfer and processing of durable, complete, and consistent data 1. Design and Implement Data Warehouses and Data Lakes: Manage appropriate data volumes and velocity while adhering to required security measures 1. Develop Processing and Analysis Algorithms: Create algorithms fit for the intended data complexity and volumes 1. Collaborate with Data Scientists: Work closely with data scientists to build and deploy machine learning models 1. Some other highly valued skills may include below: Building and maintaining scalable, reliable data pipelines Ensuring data durability, completeness, and consistency...

Role & responsibilities We are looking for a Senior Data Engineer who is responsible for the design and development of scalable data pipelines and integrations to support continual increases in data volume and complexity. Work with analytics and business teams to understand their needs, create pipelines, improve data models that feed BI and visualization tools. Key Responsibilities: Data Pipeline Development : Design, develop, and maintain robust ETL pipelines to move data from source systems to our data storage platforms. This includes data ingestion, transformation, and data quality checks to ensure accuracy and completeness. Data Modeling : Create and maintain data models and schemas that...

Title: Data Scientist Location: Chennai Job Description: Be a hands on problem solver with consultative approach, who can apply Machine Learning & Deep Learning algorithms to solve business challenges Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem Improve Model accuracy to deliver greater business impact Estimate business impact due to deployment of model Work with the domain/customer teams to understand business context , data dictionaries and apply relevant Deep Learning solution for the given business challenge Working with tools and scripts for sufficiently pre-processing the data & feature eng...

Role Overview: As a senior data pipeline engineer at Agoda, you will be working on distributed systems across multiple data centers, managing hundreds of billions of messages daily. Your primary responsibility will be to ensure data quality, integrity, and accuracy. You will have the opportunity to design new scalable systems to handle the increasing volume of data, including auditing and monitoring systems. Additionally, you will be managing projects with a small team, enhancing your ownership and leadership skills. Your role will involve solving complex problems related to managing and analyzing large amounts of data, such as schema registry, real-time data-ingestion, cross data center rep...

Job Description: Agoda is an online travel booking platform that connects travelers worldwide with a vast network of 4.7M hotels, holiday properties, flights, activities, and more. As part of Booking Holdings and with over 7,100 employees from 95+ nationalities in 27 markets, Agoda fosters a diverse and collaborative work environment. The Data department at Agoda plays a critical role in leveraging data to empower decision-making processes across the company. As a Senior Data Pipeline Engineer, you will work on distributed systems, ensuring data quality, integrity, and accuracy in processing hundreds of billions of messages daily. Key Responsibilities: - Build, administer, and scale data pip...

8+ years of experience in application maintenance and support, with a focus on data-driven systems. Experience of managing Cloud Data based applications (preferably Snowflake, Matillion or other ETL tools) especially from Support & Maintenance perspective. Experience with ITSM tools (e.g., ServiceNow, Jira) for tracking Service Requests and enhancements. Performing Root Cause analysis and taking steps to reduce recurring incidents. Collaborate with cross-functional teams including Data Engineers, Data Architects and business analysts to ensure seamless application operations. Manage application upgrades, patches, and configuration changes. Develop and maintain documentation for application a...