603 Hdfs Jobs - Page 24

As AWS Data Engineer at organization, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives You will design and maintain scalable data pipelines using AWS data analytical resources, enabling efficient data processing and analytics. Key Responsibilities: - Highly experiences in developing ETL pipelines using AWS Glue and EMR with PySpark/Scala. - Utilize AWS services (S3, Glue, Lambda, EMR, Step Functions) for data solutions. - Design scalable data models for analytics and reporting. - Implement data validation, quality, and governance ...

Job Type : Full-Time Location Gurgaon Experience – 5-10 years Role – Big Data Architect. About the Role: As a Big Data Engineer, you will play a critical role in integrating multiple data sources, designing scalable data workflows, and collaborating with data architects, scientists, and analysts to develop innovative solutions. You will work with rapidly evolving technologies to achieve strategic business goals. Must-Have Skills: 4+ year’s of mandatory experience with Big data. 4+ year’s mandatory experience in Apache Spark. Proficiency in Apache Spark, Hive on Tez, and Hadoop ecosystem components. Strong coding skills in Python & Pyspark. Experience building reusable components or framework...

Job: Software Developer (Java + Big Data) Location: Indore Years of experience: 5-7 years Requisition Description 1. Problem solving and analytical skills 2. Good verbal and written communication skills Roles and Responsibilities 1. Design and develop high performance, scale-able applications with Java + Bigdata as minimum required skill . Java, Microservices , Spring boot, API ,Bigdata-Hive, Spark, Pyspark 2. Build and maintain efficient data pipelines to process large volumes of structured and unstructured data. 3. Develop micro-services, API and distributed systems 4. Worked on Spark, HDFS, CEPh, Solr/Elastic search, Kafka, Deltalake 5. Mentor and Guide junior members

Senior Big Data Engineer Experience: 7-9 Years of Experience. Preferred location: Hyderabad Must have Skills: Bigdata, AWS cloud, Java/Scala/Python, Ci/CD Good to Have Skills: Relational Databases (any), No SQL databases (any), Microservices or Domain services or API gateways or similar, Containers (Docker, K8s, etc) Required Skills Big Data,Aws Cloud,CI/CD,Java/Scala/Python

Role Proficiency: Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others' developmental activities Outcomes: Interpret the application/feature/component design to develop the same in accordance with specifications. Code debug test document and communicate product/component/feature development stages. Validate results with user representatives; integrates and commissions the overall solution Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or creating own solu...

Job Title: Big Data Developer Project Support & Mentorship Location: Mumbai Employment Type: Full-Time/Contract Department: Engineering & Delivery Position Overview: We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent. You will work on live client projects, provide technical support, contribute to solution delivery, and serve as a go-to technical mentor for less experienced team members. Key Responsibilities: Perf...

Position : Data Engineer Experience: 4+ years Availability: Immediate joiners preferred Work Mode: Work from Office (WFO) Location: MIDC, Ghansoli, Navi Mumbai Working Days: 5 days a week (Client Location) About the Role: We are seeking a skilled and detail-oriented Data Engineer to join our growing team. The ideal candidate will have strong proficiency in Python, SQL and Databricks and experience in building and maintaining scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other engineers to ensure data availability, reliability, and quality across our systems. Required Skills & Qualifications: Experience with Databricks - Mandatory Profic...

6+ years of total IT experience 3+ years of experience with Hadoop (Cloudera)/big data technologies Knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr) Experience in designing and developing Data Pipelines for Data Ingestion or Transformation using Java Scala or Python. Experience with Spark programming (Pyspark, Scala, or Java) Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required. Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus. Hand on experience in CI/CD, Scheduling and Scripting Ensur...

About the Role As a Senior Data Engineer you’ll be a core part of our engineering team. You will bring your valuable experience and knowledge, improving the technical quality of our data-focused products. This is a key role in helping us become more mature, deliver innovative new products and unlock further business growth. This role will be part of a newly formed team that will collaborate alongside data team members based in Ireland, USA and India. Following the successful delivery of some fantastic products in 2024, we have embarked upon a data-driven strategy in 2025. We have a huge amount of data and are keen to accelerate unlocking its value to delight our customers and colleagues. You...

Warm Greetings from SP Staffing!! Role: Big Data Developer Experience Required :4 to 8 yrs Work Location : Bangalore/Chennai/Pune/Delhi/Hyderabad/Kochi Required Skills, Spark and Scala Interested candidates can send resumes to nandhini.s@spstaffing.in

Job: Data Engineer Experience: 5+ years Mandatory Skill: Python, Pyspark, Linux Shell Scripting Location: Trivandrum Required Skills & Experience: Experience with large-scale distributed data processing systems. Expertise in data modelling, testing, quality, access, and storage. Proficiency in Python, SQL , and experience with Databricks and DBT. Experience implementing cloud data technologies (GCP, Azure, or AWS). Knowledge of improving the data development lifecycle and shortening lead times. Agile delivery experience. Required Skills Python,Pyspark,Linux Shell Scripting

We're Nagarro , We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale across all devices and digital mediums, and our people exist everywhere in the world (18000+ experts across 38 countries, to be exact). Our work culture is dynamic and non-hierarchical. We are looking for great new colleagues. That is where you come in! REQUIREMENTS: Total experience 6+ years. Excellent knowledge and experience in Big Data engineer. Deep understanding of big data technologies, including Spark, Scala, AWS/Azure/GCP (Any 1), Hadoop, Hive, Maven, SQL Server . Must have experience in Hadoop, Hi...

Skills : Bigdata, Pyspark, Hive, Spark Optimization Good to have : GCP Roles and Responsibilities Skills : Bigdata, Pyspark, Hive, Spark Optimization Good to have : GCP

Were looking for a Python Developer with 3+ years of experience in AI/ML development and distributed systems. The ideal candidate is skilled in Python, understands core machine learning algorithms, AI advancements and has some experience/knowledge working with distributed computing and storage frameworks. Knowledge of backend architecture is a strong plus Role & responsibilities Preferred candidate profile - Strong Python programming skills - Understands Databases and Object Storage - Experience with ML algorithms and model development - Familiarity with distributed systems (e.g., Spark, Dask, Ray) - Exposure to large-scale data storage (e.g., S3, HDFS) - Good grasp of software engineering b...

Mandatory - Experience and knowledge in designing, implementing, and managing non-relational data stores (e.g., MongoDB, Cassandra, DynamoDB), focusing on flexible schema design, scalability, and performance optimization for handling large volumes of unstructured or semi-structured data. Mainly client needs No SQL DB, either MongoDB or HBase Data Pipeline Development: Design, develop, test, and deploy robust, high-performance, and scalable ETL/ELT data pipelines using Scala and Apache Spark to ingest, process, and transform large volumes of structured and unstructured data from diverse sources. Big Data Expertise: Leverage expertise in the Hadoop ecosystem (HDFS, Hive, etc.) and distributed ...

Cloud Data Engineer Req number: R5934 Employment type: Full time Worksite flexibility: Remote Who we are CAI is a global technology services firm with over 8,500 associates worldwide and a yearly revenue of $1 billion+. We have over 40 years of excellence in uniting talent and technology to power the possible for our clients, colleagues, and communities. As a privately held company, we have the freedom and focus to do what is right—whatever it takes. Our tailor-made solutions create lasting results across the public and commercial sectors, and we are trailblazers in bringing neurodiversity to the enterprise. Job Summary We are seeking a motivated Cloud Data Engineer that has experience in bu...

Overview Annalect is currently seeking a data engineer to join our technology team. In this role you will build Annalect products which sit atop cloud-based data infrastructure. We are looking for people who have a shared passion for technology, design & development, data, and fusing these disciplines together to build cool things. In this role, you will work on one or more software and data products in the Annalect Engineering Team. You will participate in technical architecture, design and development of software products as well as research and evaluation of new technical solutions. Responsibilities Steward data and compute environments to facilitate usage of data assets Design, build, te...

Exp-5-12 years MUST Have: End to End ETL Testing hands on and should be really strong in process work flow. MUST Have: Should be able to write Complex SQL MUST Have: UNIX or Shell scripting hands-on experience needed for ETL automation work MUST Have: Bigdata -HDFS, Hive, Good to have : Nifi pipeline Good to have : Python or Scala or Impala Position available for Pune location only

Were Hiring – Big Data Engineer (Scala + Spark) Location: Bangalore – Whitefield (Hybrid: 3 Days Office / Week) Budget: Up to 30 LPA Availability: Immediate to Max 2 Weeks Notice Must Have Skills: Scala & Spark (Mandatory) Big Data & Hadoop Ecosystem Hive, Kafka, Oozie Python Data Structures & Algorithms Design & Coding Skills CI/CD Pipelines Problem Solving & Communication Skills Strong Organizational Skills Nice to Have: Experience in distributed data processing & performance optimization Why Join Us? At Xebia, you’ll be part of a high-performance engineering team solving challenging data problems for global clients. We value innovation, ownership, and continuous learning. How to Apply: Se...

Experience Required : 10+ years overall, with 5+ years in Kafka infrastructure management and operations. Must have successfully deployed and maintained Kafka clusters in production environments, with proven experience in securing, monitoring, and scaling Kafka for enterprise-grade data streaming. Overview : We are seeking an experienced Kafka Administrator to lead the deployment, configuration, and operational management of Apache Kafka clusters supporting real-time data ingestion pipelines. The role involves ensuring secure, scalable, and highly available Kafka infrastructure for streaming flow records into centralized data platforms. Role & responsibilities Architect and deploy Apache Kaf...

Skills : Bigdata, Pyspark, Hive, Spark Optimization Good to have : GCP Roles and Responsibilities Skills : Bigdata, Pyspark, Hive, Spark Optimization Good to have : GCP

Key Responsibilities: Design, build, and optimize large-scale data processing systems using distributed computing frameworks like Hadoop, Spark, and Kafka . Develop and maintain data pipelines (ETL/ELT) to support analytics, reporting, and machine learning use cases. Integrate data from multiple sources (structured and unstructured) and ensure data quality and consistency. Collaborate with cross-functional teams to understand data needs and deliver data-driven solutions. Implement data governance, data security, and privacy best practices. Monitor performance and troubleshoot issues across data infrastructure. Stay updated with the latest trends and technologies in big data and cloud computi...

Experience Required : 10+ years overall, with 5+ years in Kafka infrastructure management and operations. Must have successfully deployed and maintained Kafka clusters in production environments, with proven experience in securing, monitoring, and scaling Kafka for enterprise-grade data streaming. Overview : We are seeking an experienced Kafka Administrator to lead the deployment, configuration, and operational management of Apache Kafka clusters supporting real-time data ingestion pipelines. The role involves ensuring secure, scalable, and highly available Kafka infrastructure for streaming flow records into centralized data platforms. Role & responsibilities Architect and deploy Apache Kaf...

Exp - 4 to 7 Yrs Loc - Bangalore or Hyderabad Posi - Permanent FTE Must Have Skills - Python, PySpark, Hadoop, Hive, Big Data Technologies, Scala, SQL, Airflow, Kafka Required Candidate profile Looking for strong Immediate Joiners or candidates with last working date in August from Hyderabad or Bangalore or South India for Big Data Developer positions for a US banking major client.

Data Pipeline Development: Design and develop efficient big data pipelines (batch as well as streaming) using Apache Spark, Trino, ensuring timely and accurate data delivery. Collaboration and Communication: Work closely with data scientists, analysts, and stakeholders to understand data requirements and perform exploratory data analysis to recommend best feature attributes, data models for AI model training and deliver highquality data solutions. Data Exploration: Analyse customer data and patterns and suggest use cases in BFSI like Instights, AI & GenAI use cases. Data Quality and Security: Ensure data quality, integrity, and security across all data platforms, maintaining robust data gov...