895 Hdfs Jobs - Page 10

KEY RESPONSIBILITIES: 5+ years of experience working in a software product development organization building modern and scalable big data applications • Expertise with functional programming, preferably Scala • Fluency in Hadoop technologies and big data processing technologies, e.g. Spark, YARN, HDFS, Oozie, Hive • Experience implementing RESTful web services in Java, Scala, Python, or similar languages. • Experience with NoSQL and SQL databases • Experience in information retrieval and machine learning

Job Description: We are seeking a skilled Data Engineer with extensive experience in the Cloudera Data Platform (CDP) to join our dynamic team. The ideal candidate will have over four years of experience in designing, developing, and managing data pipelines, and will be proficient in big data technologies. This role requires a deep understanding of data engineering best practices and a passion for optimizing data flow and collection across a diverse range of sources. Required Skills and Qualifications: Experience: 4+ years of experience in data engineering, with a strong focus on big data technologies. Cloudera Expertise: Proficient in Cloudera Data Platform (CDP) and its ecosystem, includin...

Job Summary: We are seeking a highly skilled Data Engineer with a strong background in Python. The ideal candidate will have 7 10 years of experience in data engineering, with a proven track record of designing, building, and maintaining scalable data pipelines and architectures. You will play a crucial role in transforming raw data into actionable insights, enabling data driven decision making across the organization. Responsibilities: Design, develop, and maintain robust data pipelines and ETL processes to support data ingestion and transformation. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high quality data solutions. Imp...

Role & responsibilities Mandatory Skills Python | PySpark | SQL | Traditional ML | MLOps Job Summary We are looking for an AI/ML Engineer who can design, build, and maintain scalable data pipelines and MLOps systems. This role involves performing experiments, statistical analysis, EDA, feature engineering, and productionizing ML models. You will be responsible for optimizing training and inference, enabling the organization to build, deploy, and scale AI/ML-driven solutions. Roles & Responsibilities Build and maintain data ingestion, transformation, and feature engineering pipelines Conduct experiments, statistical analysis & EDA to generate insights for model improvement Ensure pipelines ar...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and Azure Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark and Hive, Hbase or other NoSQL databases on Azure Cloud Data Platform or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing strea...

Job Summary We are seeking a highly skilled Sr. Developer with 6 to 10 years of experience to join our team in a hybrid work model. The ideal candidate will have expertise in Databricks SQL Databricks Delta Lake Databricks Workflows and PySpark. This role involves developing and optimizing data solutions to drive business insights and efficiencies. The position is a day shift role with no travel requirements. Responsibilities Develop and implement data solutions using Databricks SQL to enhance business intelligence capabilities. Optimize data processing workflows with Databricks Delta Lake to ensure data integrity and performance. Design and manage Databricks Workflows to automate data pipel...

Meet the Team Cisco ThousandEyes is a Digital Assurance platform that empowers organizations to deliver flawless digital experiences across every network - even the ones they don't own. Powered by AI and an unmatched set of cloud, internet and enterprise network telemetry data, ThousandEyes enables IT teams to proactively detect, diagnose, and remediate issues - before they impact end- user experiences. ThousandEyes is deeply integrated across the entire Cisco technology portfolio and beyond, helping customers deploy at scale while also delivering AI-powered assurance insights within Cisco's leading Networking, Security, Collaboration, and Observability portfolios. Your Impact As a Site Reli...

Greetings from TCS! TCS is hiring for Pyspark developer Desired Experience Range: 4 to 8 Years Job Location: Chennai / Mumbai / Pune Key Responsibilities: Develop, optimize, and maintain big data pipelines using PySpark on distributed computing platforms. Design and implement ETL workflows for ingesting, processing, and transforming large datasets in Hive. Work with structured and unstructured data sources to ensure efficient data storage and retrieval. Optimize Hive queries and Spark jobs for performance, scalability, and cost efficiency. Implement best practices for data engineering, including data governance, security, and compliance. Monitor, troubleshoot, and enhance data workflows to e...

Introduction In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology. Your Role And Responsibilities As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and Azure Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, ...

TCS is Hiring Data Engineer (Spark/Scala)!! Location: Hyderabad, Bangalore, Chennai, Pune, Gurugram Experience: 7 - 10 yrs (Accurate) Mode of Interview: Virtual Date of Interview: 30th October,2025 Notice Period: 0-30 Days / Immediate Joiners are preferred Required Technical Skill : Scala/Spark , Hadoop , Hive Responsibilities: Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Experience working on Data Engineering projects Good Understanding of SQL Good understanding of Unix and HDFS commands Communication Experience working on Data Analytics and Pyspark Exposure to Tableau & SAS Kindly attach your updated resume Thanks & Regards Malleswari M

7+ years of experience in Big Data with strong expertise in Spark and Scala Mandatory Skills: Big Data Primarily Spark and Scala Strong Knowledge in HDFS, Hive, Impala with knowledge on Unix , Oracle, Autosys, Good to Have : Agile Methodology and Banking Expertise Strong Communication Skills Not limited to Spark batch, need Spark streaming experience No SQL DB Experience : HBase/Mongo/Couchbase

Job Description: About Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities and shareholders every day. One of the keys to driving Responsible Growth is being a great place to work for our teammates around the world. We're devoted to being a diverse and inclusive workplace for everyone. We hire individuals with a broad range of backgrounds and experiences and invest heavily in our teammates and their families by offering competitive benefits to support their physical, emotional, and financial well-being. Bank of...

Job description Role & responsibilities Test ETL Processes, ensuring accurate extraction, transformation and loading of data. Validation source to target mappings, transformation and business rules. Perform data completeness, accuracy and consistency checks. Strong on Data Engineering domain, should be aware or handed large Data Volumes testing , Big Data, ETL testing , worked on Hadoop/HDFS related apps, Python Programming is a plus.

As a Big Data Developer at Unison Consulting Pte Ltd, you will play a crucial role in designing, developing, and maintaining big data solutions to enhance data-driven decision making and analytics capabilities for our clients. You will leverage cutting-edge technologies and platforms to work with large volumes of structured and unstructured data. Responsibilities: - Design, develop, and maintain big data solutions utilizing technologies like Hadoop, HDFS, Spark, and Hive. - Ingest, process, and analyze substantial amounts of structured and unstructured data from various sources. - Develop and optimize ETL processes to facilitate efficient data extraction, transformation, and loading. - Imple...

Role: Big Data Developer Experience: 6 Location: Chennai/Gurgaon Required Technical Skill Set: GCP , HDFS, Unix Script, Autosys ,Hive. Must Have: Hands on experience on Spark framework, Unix scripting, Hive queries, writing UDF in Hive. Theortical knowledge and POC alone will not suffice. Good knowledge in Software Development Life Cycle and Project Development Lifecycle. Associate should be able to work independently and should have strong debugging skill in both Hive and Spark. Associate should have experience developing large scale systems, experience debugging and performance tuning, excellent software design skills, communication skills and ability to work with client partners.

About the Team! DE team at HYD collaborates with other engineering/product/analyst teams across tech sites to collectively accomplish OKRs to take Uber forward. We constantly enrich our data layer to effectively deal with the next generation of products which are a result of Uber's big bold bets. We design and build data pipelines to schedule & orchestrate a variety of tasks such as extract, cleanse, transform, enrich & load data as per the business needs. We serve the data insights at depth to various teams at Uber-like business analysts, data analysts, data science and business stakeholders to take strategic decisions, train DS models, perform health checks etc in order to grow and support...

As a Software Developer you will work in a constantly evolving environment, due to technological advances and the strategic direction of the organization you work for. You will create, maintain, audit, and improve systems to meet particular needs, often as advised by a systems analyst or architect, testing both hard and software systems to diagnose and resolve system faults. The role also covers writing diagnostic programs and designing and writing code for operating systems and software to ensure efficiency. When required, you will make recommendations for future developments Benefits of Joining Us Challenging Projects : Work on cutting-edge projects and solve complex technical problems. Ca...

Requirements Experience At least 5 years of experience in AWS based projects. Technical skills Proficiency in Python and PySpark for data engineering tasks. Big Data Strong knowledge of Big Data technologies and data warehousing concepts. AWS services Experience with AWS Data Engineering stack, including S3, RDS, Athena, Glue, Lambda, and Step Functions. SQL Strong SQL skills for data manipulation and querying. CI CD Experience with CI CD tools like Terraform and Git Actions. Soft skills Good communication skills and ability to work in a multicultural team. Design and implement data pipelines Develop ETL jobs to ingest and move data within the AWS environment using tools like AWS Glue Data s...

Job Summary: We are seeking a skilled and experienced PySpark Developer with 3 to 5 years of hands-on experience in building and optimizing distributed data pipelines , data processing workflows , and data validation using Apache Spark with Python (PySpark) . The ideal candidate should have strong analytical and problem-solving skills, and be proficient in handling large-scale datasets, performing transformations, and ensuring data quality across distributed systems. Key Responsibilities: Develop, maintain, and optimize PySpark scripts for distributed data processing, cleansing, and validation. Work extensively with Spark Data Frames for data wrangling, aggregation, and transformation. Desig...

About The Role Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : Apache Hadoop Good to have skills : NA Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will design, build, and configure applications to meet business process and application requirements. A typical day involves collaborating with team members to understand project needs, developing application features, and ensuring that the applications function seamlessly within the existing infrastructure. You will also ...

6-7+ years of Experience in Hadoop or any Cloud Bigdata components (specific to the Data Engineering role), Hadoop hands on (airflow, oozie, hive, hdfs, sqoop, pig, map, reduce ) 4+ years of Exp in Spark (Spark Batch, Streaming, Mlibetc). Candidates should possess proficiency in utilizing the Apache Spark framework. 6-7+ years of experience in Python programming language. 4+ years of experience with pyspark data transformation(json,csv,rdbms,stream) pipeline design , development and deployment with kubernate/onprem platform (not cloud based). 2+ years of experience in designing and implementing data workflows with Apache Airflow. Kafka or equivalent Cloud Bigdata components (specific to the ...

Job Role: Data Engineer Exp: 5+ Years Loc: Bangalore Deep Hadoop ecosystem knowledge: Spark, HDFS, Hive, Iceberg. Batch and streaming ETL pipeline design scalability and reliability Python & Unix proficiency, materialized views, data structures/algorithms. Data warehouse security, compliance, and operational excellence. Automated pipeline design & optimization, platform collaboration Monitoring and logging: Prometheus, Grafana, Jenkins/GitHub Actions Testing frameworks: JUnit/pytest , code cleanliness Skills -Big data solutions, Hadoop data warehouse, analytics infrastructure. Building scalable pipelines, system optimization, architectural reviews. Spark, HDFS, Hive, Iceberg, Kafka, Flink, P...

Skills - Hadoop , Hive , HDFS , ETL Processes , CI/CD Data Engineer Job Description: As a Data Engineer on our team, you will work on our Hadoop-based data warehouse, contributing to scalable and reliable big data solutions for analytics and business insights. This is a hands-on role focused on building, optimizing, and maintaining large data pipelines and warehouse infrastructure. Key Responsibilities Design, develop, and maintain robust data pipelines in Hadoop and related ecosystems, ensuring data reliability, scalability, and performance. Implement data ETL processes for batch and streaming analytics requirements. Optimize and troubleshoot distributed systems for ingestion, storage, and ...

As a Hadoop Big Data Trainer at Wing Educations, you will play a crucial role in leading engaging training sessions on Hadoop and Big Data concepts for students and professionals. Your responsibilities will include: - Preparing training materials to ensure comprehensive coverage of Hadoop and Big Data tools such as HDFS, MapReduce, Hive, HBase, and Pig - Customizing content to cater to different learning needs and levels of expertise - Conducting hands-on demos and exercises to facilitate practical understanding of the concepts - Staying updated with the latest industry trends in Hadoop and Big Data technologies - Assessing trainees" progress and providing constructive feedback to help them ...

As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution. Your primary responsibilities include: Lead the design and construction of new solutions using the latest technologies, always looking to add business value and meet user requirements. Strive for continuous improvements by testing the build solution and working under an agile framework. Discover and implement the latest technologies trends to maximize and build creative solutions Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Ex...