575 Hdfs Jobs - Page 11

Unlock your potential with Dassault Systmes, a global leader in Scientific Software Engineering as a Big Data Engineer in Pune, Maharashtra! Role Description & Responsibilities: - Data Pipeline Development: Design, develop, and maintain robust ETL pipelines for batch and real-time data ingestion, processing, and transformation using Spark, Kafka, and Python. - Data Architecture: Build and optimize scalable data architectures, including data lakes, data marts, and data warehouses, to support business intelligence, reporting, and machine learning. - Data Governance: Ensure data reliability, integrity, and governance by enabling accurate, consistent, and trustworthy data for decision-making. - ...

This position falls under the ICG TTS Operations Technology (OpsTech) Group, focusing on assisting in the implementation of a next-generation Digital Automation Platform and Imaging Workflow Technologies. The ideal Candidate should have relevant experience in managing development teams within the distributed systems Eco-System and must exhibit strong teamwork skills. The candidate is expected to possess superior technical knowledge of current programming languages, technologies, and leading-edge development tools. The primary objective of this role is to contribute to applications, systems analysis, and programming activities. As a Lead Spark Scala Engineer, the candidate should have hands-o...

As a Data Engineer Trainer/Big Data Trainer, you will be responsible for imparting knowledge and training on various technical aspects related to data engineering and big data. Your key responsibilities will include expertise in Data Mining and ETL Operations/Tools. It is crucial to have a deep understanding and knowledge of HDFS, Hadoop System, Map Reduce, RDD, Spark DataFrame, PySpark along with related concepts. You should also have experience in using Business Intelligence Tools such as Tableau, Power BI, and Big Data Frameworks like Hadoop and Spark. Proficiency in Pig, Hive, Sqoop, and Kafka is essential for this role. Knowledge of AWS and/or Azure, especially with Big Data Stack, will...

Big Data Development Administration.AWS EMR clusters setup, scaling, tuningSpark (PySpark/Scala), Hive, HDFS, YARN.SQL Redshift/ SnowflakeAWS services: S3, IAM, Lambda, Glue, CloudWatch, Step Functions.Linux/Unix(Airflow, Oozie, Control-M, Autosys).

Notice Period : Immediate to 30 days Mandatory Scala and Python Apache Spark (batch & streaming) must! Deep knowledge of HDFS internals and migration strategies. Experience with Apache Iceberg (or similar table formats like Delta Lake / Apache Hudi) for schema evolution, ACID transactions, and time travel. Running Spark and/or Flink jobs on Kubernetes (e.g., Spark-on-K8s operator, Flink-on-K8s). Experience with distributed blob storages like Ceph or AWS S3 and similar Building ingestion, transformation, and enrichment pipelines for large-scale datasets. Infrastructure-as-Code (Terraform, Helm) for provisioning data infrastructure. Ability to work independently while guiding juniors. Nice to ...

As a DataOps Lead, you will be responsible for managing, design highly-scalable and Available solution for data pipelines that provides the foundation for collecting, storing, modeling, and analyzing massive data sets from multiple channels. This position reports to Devops Architect. Responsibilities: Align Sigmoid with key Client initiatives Interface daily with customers across leading Fortune 500 companies to understand strategic requirements Connect with VP and Director level clients on a regular Travel to client locations Ability to understand business requirements and tie them to technologysolutions Strategically support Technical Initiatives Design, manage & deploy highly scalable and...

As a data engineer at our company, you will play a crucial role in designing and implementing large-scale systems, particularly focusing on complex data pipelines. You will be responsible for driving projects from the initial stages to production, collaborating with stakeholders, analysts, and scientists to gather requirements and transform them into a data engineering roadmap. Your ability to communicate effectively, work well within a team, and showcase strong technical skills will be key in this role. Your primary responsibilities will include collaborating with various teams across different tech sites to achieve Objectives and Key Results (OKRs) that propel our company forward. You will...

Job Description As aSenior Software Engineer in our team, you work with large scale manufacturing data coming from our globally distributed plants. You will focus on building efficient, scalable & data-driven applications that - among other use cases - connect IoT devices, pre-process, standardize or enrich data, feed ML models or generate alerts for shopfloor operators. The data sets produced by these applications - whether data streams or data at rest - need to be highly available, reliable, consistent and quality-assured so that they can serve as input towide range of other use cases and downstream applications. We run these applications on a hybrid data platform - Azure Databricks and a ...

Role - Big Data Tester (Immediate Joiner) Location - Gurugram/ Bengaluru Duration - Full Time Roles and Responsibilities Design, develop, and execute comprehensive test plans for big data solutions using Hadoop Distributed File System (HDFS), Hive, and SQL. Collaborate with cross-functional teams to identify requirements and ensure seamless integration of ETL processes. Develop automated testing scripts using Python or other programming languages to improve efficiency and reduce manual errors. Analyze complex data sets to identify trends, patterns, and anomalies, providing insights that inform business decisions. Participate in Agile development methodologies such as Scrum to deliver high-qu...

**ONLY FOR IMMEDIATE JOINERS** Position: Big Data Tester Experience: 4+ Years Location: ( Gurugram / Bangalore ) Joining: Immediate Joiner Job Summary: We are seeking an experienced Big Data Tester with a strong background in HDFS, Hive, ETL testing, and SQL. The ideal candidate will have a experience in Big Data ecosystems , ETL/SQL testing , and in the healthcare domain . You will be responsible for validating data pipelines, ensuring data quality, and verifying ETL processes in a big data environment. Key Responsibilities: Design and execute comprehensive test plans and test cases for Big Data applications and ETL processes. Validate large-scale datasets stored in HDFS using HiveQL and ot...

Join us as a Senior Developer at Barclays, where you will play a crucial role in supporting the successful delivery of Location Strategy projects while adhering to plan, budget, quality, and governance standards. Your primary responsibility will be to drive the evolution of our digital landscape, fostering innovation and excellence. By leveraging cutting-edge technology, you will lead the transformation of our digital offerings, ensuring unparalleled customer experiences. To excel in this role as a Senior Developer, you should possess the following experience and skills: - Solid hands-on development experience with Scala, Spark, Python, and Java. - Excellent working knowledge of Hadoop compo...

As a Data Engineer II at Media.net, you will be responsible for designing, executing, and managing large and complex distributed data systems. Your role will involve monitoring performance, optimizing existing projects, and researching and integrating Big Data tools and frameworks as required to meet business and data requirements. You will play a key part in implementing scalable solutions, creating reusable components and data tools, and collaborating with teams across the company to integrate with the data platform efficiently. The team you will be a part of ensures that every web page view is seamlessly processed through high-scale services, handling a large volume of requests across 5 m...

Common Skills - SQL, GCP BQ, ETL pipelines using Pythin/Airflow, Experience on Spark/Hive/HDFS, Data modeling for Data conversion Resources (4) Prior experience working on a conv/migration HR project is additional skill needed along with above mentioned skills Common Skils - SQL, GCP BQ, ETL pipelines using Pythin/Airflow, Experience on Spark/Hive/HDFS, Data modeling for Data conversion Resources (4) Prior experience working on a conv/migration HR project is additional skill needed along with above mentioned skills Data Engineer - Knows HR Knowledge , all other requirement from Functional Area given by UBER

Experience in automation, Java with selenium, rest assured is must. Experience with HDFS, Linux Commands Experience in Testing large data set which includes large extracts , Hive to Hive etc. Good knowledge of different Testing methodologies, concepts Experience in debugging application log files. Familiarity with DevOps (CI/CD) Mandatory Competencies QA/QE - QA Automation - Automation of Rest, Web / SOAP UI Services QA/QE - QA Automation - Selenium Beh - Communication and collaboration Big Data - Big Data - HDFS

The Applications Development Intermediate Programmer Analyst position at our organization involves working at an intermediate level to assist in the development and implementation of new or updated application systems and programs in collaboration with the Technology team. Your main responsibility will be to contribute to application systems analysis and programming activities. You will be expected to utilize your knowledge of applications development procedures and concepts, as well as basic knowledge of other technical areas, to identify and define necessary system enhancements. This includes using script tools, analyzing code, and consulting with users, clients, and other technology group...

Were Hiring: Tech Lead – Big Data Technologies Location: Pune | Work Mode: Work from Office (WFO) Company: Leading MNC | Immediate Joiners Preferred Are you a Big Data expert with 6–9 years of experience and a passion for leading high-impact projects? We’re looking for a Tech Lead skilled in: Hadoop, Spark, Hive, Kafka, Sqoop, Oozie, Flume Java/Scala, SQL/NoSQL (PostgreSQL, MongoDB) System monitoring tools (Grafana, Ganglia), scripting & automation Role involves end-to-end ownership of data pipeline architecture, code reviews, performance tuning, and leading a talented team.

Hadoop Support Engineer Bengaluru, KA - WFO We are seeking a dedicated Hadoop Support Engineer to provide comprehensive support and on-call services for our Hadoop infrastructure. This role is pivotal in ensuring the continuous operation and stability of our Hadoop clusters, addressing incidents promptly, and supporting end-users with technical queries. The ideal candidate will possess strong Hadoop administration skills, effective troubleshooting capabilities, and excellent communication skills to interact with various stakeholders. Key Responsibilities: On-Call Support: Serve as a primary point of contact in a rotating on-call schedule, providing 24/7 support to swiftly address and resolve...

About the Role: 08 The Team: As a member of the S&P Global Market Intelligence Technology team, you will work with a group of intelligent and ambitious engineers. Our software engineers are involved in the full product life cycle, from design through release. You will be expected to participate in the design review process, write high-quality code, and work with a dedicated team of QA Analysts, Business Analysts, and Infrastructure Teams. The Impact: The person in this position will be responsible for creating technical designs, developing high-performing web application software, and ensuring quality through a robust testing process. Software Engineers at S&P Global work with the latest tec...

Role & responsibilities Machine Learning A machine learning expert should not only be able delve into concepts such as data analytics, big data, artificial intelligence, machine learning and deep learning, etc., Also help other team member in by challenging the thought process and bringing in innovative thinking. Ensuring the execution through collaborative working and problem solving. This role will require areas to be identified for value creation that will benefit from new technology, technique or algorithm deployment. The candidate is expected to possess a strong technical and business competency with knowledge of current industry Machine Learning standards, structures, and best practice...

At Tredence AWS Data Engineer designs, builds, and maintains scalable data pipelines and infrastructure using AWS services like Glue, S3, and Redshift, focusing on data ingestion, transformation (ETL), modeling, and quality. They write SQL and Python code, orchestrate workflows with tools like Airflow, and collaborate with data scientists and analysts, aiming to ensure data is accurate, accessible, and optimized for analysis and business decision-making. Role - AWS Data Engineer Years of Experience: 2~5 years Roles and responsibilities The day-to-day work of a junior-to-mid AWS data engineer involves a mix of development, monitoring, and collaboration. Pipeline development: Designing, buildi...

Overview: We are seeking a highly motivated and detail-oriented individual to join our team as a Big Data Developer. This role requires a dynamic professional who can adapt to evolving business needs and drive value through their expertise. Key Responsibilities: Provide support and expertise in the domain of Big Data Developer. Collaborate with cross-functional teams to achieve business goals. Ensure timely delivery of services and maintain high-quality standards. Required Qualifications: Proven experience in a relevant field or position. Strong understanding of the responsibilities and tools associated with the role. Excellent problem-solving and communication skills. Preferred Qualificatio...

Enphase Energy is a global energy technology company and leading provider of solar, battery, and electric vehicle charging products. Founded in 2006, we transformed the solar industry with our revolutionary microinverter technology, turning sunlight into a safe, reliable, resilient, and scalable source of energy to power lives. Our Enphase Energy System assists individuals in making, using, saving, and selling their own power. We are proud to be one of the fastest-growing and innovative clean energy companies worldwide, with approximately 68 million products installed in over 145 countries. We are currently seeking an Engineer to join our Storage business unit's Data Analytics and Engineerin...

As a Data Engineer, you will be responsible for building highly scalable, fault-tolerant distributed data processing systems that handle hundreds of terabytes of data daily and manage petabyte-sized data warehouses and Elasticsearch clusters. Your role will involve developing quality data solutions and simplifying existing datasets into self-service models. Additionally, you will create data pipelines that enhance data quality and are resilient to unreliable data sources. You will take ownership of data mapping, business logic, transformations, and data quality, and engage in low-level systems debugging and performance optimization on large production clusters. Your responsibilities will als...

Key Skills: Hadoop, Spark, Python, SQL, Scala, HDFS, Hive, Kafka, HBase Roles & Responsibilities: Design, develop, and maintain large-scale data processing systems using Hadoop and Spark. Write optimized Spark jobs using Scala to ensure efficient data processing. Implement and manage data pipelines and workflows using HDFS, Hive, Kafka, and HBase. Collaborate with cross-functional teams to understand data requirements and deliver solutions. Ensure high performance and responsiveness of applications by optimizing data processing tasks. Stay updated with the latest trends and technologies in big data and distributed computing. Experience Requirement: 5 -8 years of experience working with Hadoo...

Your key responsibilities Work as a project manager to Lead the design and evolution of our large-scale data lakehouse architecture, ensuring it is scalable, reliable, and cost-effective. Provide technical leadership and mentorship to data engineers and analysts. Collaborate closely with Software Engineers, and business stakeholders to understand requirements and deliver effective data solutions. Experience in data modelling, data mapping, data profiling and meta data management. Architect and tune high-performance data processing pipelines. Identify and resolve complex performance issues in distributed computing environments involving Spark execution, Hive query optimization, and Iceberg me...