24 Apache Hudi Jobs

The objectives of a Data Engineer typically revolve around designing, implementing, and managing robust data architectures and infrastructures. The Data Engineer will play a key role in driving the organization's data strategy, ensuring the efficient flow and storage of data, and supporting data-driven decision-making processes. Duties & Responsibilities Data Model Design: Develop and maintain a scalable and efficient data models and schemas that support the storage, processing, and retrieval of data. Data Integration: Implement robust ELT (Extract, Load, Transform) processes to integrate data from various sources into a unified and consistent format. Ensure seamless data flow between differ...

Job Summary We are seeking an experienced Data Lake Expert to design, implement, and optimize enterprise-grade data lake and data warehouse systems using modern open-source technologies such as Apache Iceberg, Apache Hudi, Delta Lake, and Apache Spark . The ideal candidate will have deep expertise in data ingestion, transformation, storage optimization, and governance across structured and unstructured data sources. This role involves architecting and managing scalable data platforms that enable real-time analytics, AI/ML pipelines, and regulatory compliance . Key Responsibilities 1. Data Lake Architecture & Design Architect and implement scalable data lakehouse solutions using Apache Iceber...

We are looking for a skilled Data Engineer with a strong background in PostgreSQL, SQL, and Python to join our dynamic team in Kochi. The ideal candidate will be responsible for building and optimising data pipelines, ensuring seamless data flow across systems, and supporting our analytics and product teams with reliable data solutions. Job Title: Data Engineer Company: Pumex Infotech Pvt. Ltd. Experience: 5 to 10 Years Educational qualification: B Tech/ M Tech/ BCA/MCA (Computer Science) or related field. Location : InfoPark, Kochi Notice Period: Immediate Joiners Responsibilities: Design, develop, and maintain robust ETL/ELT data pipelines using tools such as Airflow or dbt . Optimize Post...

Big Data DeveLoper Responsibilities Design and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph). Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment. Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats. Support migration strategy definition by providing technical input and identifying risks. Mentor junior developers and review their code / design decisions. Collaborate with platform engineers, cloud architects, and product stakeholders to align technical implementation with project goals. Troubleshoot complex distributed system issues in data ...

Big Data DeveLoper Responsibilities Design and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph). Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment. Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats. Support migration strategy definition by providing technical input and identifying risks. Mentor junior developers and review their code / design decisions. Collaborate with platform engineers, cloud architects, and product stakeholders to align technical implementation with project goals. Troubleshoot complex distributed system issues in data ...

Job Title: Data Engineer Experience Required: Minimum 12 Years Location: Vapi, Gujarat Employment Type: Full-time About the Role We are seeking a skilled and motivated Data Engineer to join our growing technology team. The role involves building and maintaining scalable, reliable, and secure data infrastructure to support analytics, data-driven decision-making, and AI/ML pipelines. You'll work with diverse data types and modern data platforms to design efficient data pipelines and ensure smooth data flow across systems. Key Responsibilities Design, develop, and maintain robust ETL/ELT pipelines for structured and unstructured data using tools like Apache NiFi, Airflow, or Dagster. Build stre...

Designation: Senior Engineer Data Science and Engineering Location: 22nd Floor Tower C Building No - 5 DLF Epitome Gurgaon Haryana. Tbo.com(www.tbo.com) TBO is a global platform that aims to simplify all buying and selling travel needs of travel partners across the world. The proprietary technology platform aims to simplify the demands of the complex world of global travel by seamlessly connecting the highly distributed travel buyers and travel suppliers at scale. The TBO journey began in 2006 with a simple goal to address the evolving needs of travel buyers and suppliers, and what started off as a single product air ticketing company, has today become the leading B2A (Business to Agents) tr...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Principal Data Lake Engineer at BairesDev We are seeking a Principal Data Lake Engineer with deep expertise in data lake architecture, schema-on...

Overview The Software Engineer is an integral part of our engineering team, responsible for design, implementation, testing and maintaining of innovative software products. The successful candidate must be able to quickly learn the existing systems and components, actively participate in expanding the current systems with new functionality, as well as develop new components using big data and other emerging technologies. We are looking for people who have strong knowledge of Java frameworks, experience in developing server-side and/or based applications. Responsibilities Participate in defining product level architecture and be responsible for designing component level architecture. Take own...

We are seeking an experienced and highly motivated Senior Data Engineer to join our dynamic team. The ideal candidate will play a critical role in designing, developing, and maintaining high-performance data infrastructure and pipelines. As a Senior Data Engineer, you will be responsible for ensuring the scalability, performance, and reliability of data systems that support business intelligence, analytics, and data science initiatives across the organization. You will work in a collaborative environment with cross-functional teams to translate business needs into technical solutions and help shape the organizations overall data strategy. This role offers an excellent opportunity to work wit...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success . Senior Data Lake Engineer at BairesD ev We are seeking a Senior Data Lake Engineer with expertise in data lake architecture, schema-on-read, da...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Senior Data Lake Engineer at BairesDev We are seeking a Senior Data Lake Engineer with expertise in data lake architecture, schema-on-read, data...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Principal Data Lake Engineer at BairesDev We are seeking a Principal Data Lake Engineer with deep expertise in data lake architecture, schema-on...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Data Lake Engineer at BairesDev We are seeking a Data Lake Engineer with expertise in data lake architecture, schema-on-read, data governance, a...

Description Introduction : We are on a mission to rid the world of bad customer service by mobilizing the way help is delivered. Today's consumers want an always-available customer service experience that leaves them feeling valued and respected. Helpshift helps B2B brands deliver this modern customer service experience through a mobile-first approach. We have changed how conversations take place, moving the conversation away from slow, outdated email and desktop experience to an in-app chat experience that allows users to interact with brands in their time. Through our market leading chatbots and automations, we help brands deliver instant and rapid resolutions. Because agents play a key ro...

About The Team/Role As a Senior Staff SWE of Data Lake House Engineering, this technical leader will be responsible for the design and implementation of the Data Lake house platform (supporting both Data Engineering and Data Lake house applications), Data Engineering pipeline productionalization (E2E data pipelines, model development, deployment, monitoring, refresh, etc), Data Lake house, and related technology, systems, and tools. This technologist will drive the technology development and architecture in these areas to ensure the platforms, systems, tools, models, and services generated from this team are technically sound with strong security, quality, reliability, usability, scalability...

Skills Required: Familiarity with data processing engines such as Apache Spark, Flink, or other big data tools. Design, develop, and implement robust data lake architectures on cloud platforms (AWS/Azure). Implement streaming and batch data pipelines using Apache Hudi, Apache Hive, and cloud-native services like AWS Glue, Azure Data Lake, etc. Architect and optimize ingestion, compaction, partitioning, and indexing strategies in Apache Hudi. Develop scalable data transformation and ETL frameworks using Python, Spark, and Flink. Work closely with DataOps/DevOps to build CI/CD pipelines and monitoring tools for data lake platforms. Ensure data governance, schema evolution handling, lineage tra...

Roles and Responsibilities : Design, build, and optimize scalable ETL pipelines using Apache Airflow or similar frameworks to process and transform large datasets efficiently. Utilize Spark (PySpark), Kafka, Flink, or similar tools to enable distributed data processing and real-time streaming solutions. Deploy, manage, and optimize data infrastructure on cloud platforms such as AWS, GCP, or Azure, ensuring security, scalability, and cost-effectiveness. Design and implement robust data models, ensuring data consistency, integrity, and performance across warehouses and lakes. Enhance query performance through indexing, partitioning, and tuning techniques for large-scale datasets. Manage cloud-...

AWS Storage & Compute Services S3: Strong understanding of bucket policies, lifecycle rules & data organization Glue: ETL jobs, crawlers & data cataloging Athena: Querying data directly from S3 using SQL EMR: Big data processing with Spark/Hadoop Lakehouse Technologies Apache Hudi: Hands-on knowledge of transactional data lakes & schema evolution Data Cataloging & Governance AWS Glue Data Catalog Integration with Lake Formation Security & Compliance IAM roles & policies Encryption strategies Data masking & tokenization Locations : Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune,Remote

As a Senior Staff Software Engineer in Data Lake House Engineering, you will play a crucial role in designing and implementing the Data Lake house platform, supporting both Data Engineering and Data Lake house applications. Your responsibilities will include overseeing Data Engineering pipeline productionalization, end-to-end data pipelines, model development, deployment, monitoring, refresh, etc. Additionally, you will be involved in driving technology development and architecture to ensure the platforms, systems, tools, models, and services meet the technical standards for security, quality, reliability, usability, scalability, performance, efficiency, and operability to meet the evolving ...

We are looking for an experienced Senior Data Engineer to lead the development of scalable AWS-native data lake pipelines with a strong focus on time series forecasting and upsert-ready architectures. This role requires end-to-end ownership of the data lifecycle, from ingestion to partitioning, versioning, and BI delivery. The ideal candidate must be highly proficient in AWS data services, PySpark, versioned storage formats like Apache Hudi/Iceberg, and must understand the nuances of data quality and observability in large-scale analytics systems. Role & responsibilities Design and implement data lake zoning (Raw Clean Modeled) using Amazon S3, AWS Glue, and Athena. Ingest structured and uns...

Job Title: Senior Data Engineer (4-6 Years Experience) Location: Kotak Life HO Department: Data Science & Analytics Employment Type: Full-Time About the Role: We are seeking a highly skilled Data Engineer with 4-6 years of hands-on experience in designing and developing scalable, reliable, and efficient data solutions. The ideal candidate will have a strong background in cloud platforms (AWS or Azure), experience in building both batch and streaming data pipelines, and familiarity with modern data architectures including event-driven and medallion architectures. Key Responsibilities: .Design, build, and maintain scalable data pipelines (batch and streaming) to process structured and unstruct...

Your day at NTT DATA We are seeking an experienced Data Engineer to join our team in delivering cutting-edge Generative AI (GenAI) solutions to clients. The successful candidate will be responsible for designing, developing, and deploying data pipelines and architectures that support the training, fine-tuning, and deployment of LLMs for various industries. This role requires strong technical expertise in data engineering, problem-solving skills, and the ability to work effectively with clients and internal teams. What youll be doing Key Responsibilities: Design, develop, and manage data pipelines and architectures to support GenAI model training, fine-tuning, and deployment Data Ingestion an...

Data Engineer Experience: 3 - 5 Years Exp Salary : Upto INR 35 Lacs per annum Preferred Notice Period : Within 30 Days Shift : 10:30AM to 7:30PM IST Opportunity Type: Remote Placement Type: Permanent (*Note: This is a requirement for one of Uplers' Clients) Must have skills required : Apache Airflow, Spark, AWS, Kafka, SQL Good to have skills : Apache Hudi, Flink, Iceberg, Azure, GCP Nomupay (One of Uplers' Clients) is Looking for: Data Engineer who is passionate about their work, eager to learn and grow, and who is committed to delivering exceptional results. If you are a team player, with a positive attitude and a desire to make a difference, then we want to hear from you. Role Overview De...