575 Hdfs Jobs - Page 6

As a Data Engineer, your primary responsibility will be to design and build production data pipelines within a hybrid big data architecture. You will use languages such as Scala, Python, and Talend to create efficient pipelines from data ingestion to consumption. Additionally, you will contribute to the entire data life-cycle, focusing on coding, debugging, and testing processes. Troubleshooting and optimizing ETL pipelines will also be a key aspect of your role, along with thorough documentation of each process. - Design and build production data pipelines using Scala, Python, and Talend within a hybrid big data architecture. - Gather and address technical and design requirements. - Refacto...

As a DevOps Administrator, you will be responsible for working with the latest cloud-native and distributed data platforms. Your role will involve administering Hadoop, ensuring the performance, reliability, and optimization of big data clusters, and recommending resources for deploying and optimizing big data technologies. If you are passionate about big data and leveraging cutting-edge technology to deliver concrete business outcomes, we are looking forward to your application. **Key Responsibilities:** - Install, manage, and configure big data clusters - Monitor the performance of distributed systems and middleware applications - Configure and optimize the Hadoop environment - Troubleshoo...

As a Technical Support Engineer for Big Data applications and systems in a Hadoop ecosystem, your responsibilities will include: - Providing technical support and troubleshooting for Big Data applications and systems built on the Hadoop ecosystem. - Monitoring system performance, analyzing logs, and identifying potential issues before they impact services. - Collaborating with engineering teams to deploy and configure Hadoop clusters and related components. - Assisting in maintenance and upgrades of Hadoop environments to ensure optimum performance and security. - Developing and maintaining documentation for processes, procedures, and system configurations. - Implementing data backup and rec...

As a Lead Data Engineer in our organization, you will play a crucial role in the data engineering team focused on on-premise solutions. Your responsibilities will include: - Designing, developing, and maintaining robust, scalable, and high-performance data pipelines and integration solutions. - Managing and optimizing data storage in Hadoop Distributed File System (HDFS). - Designing and implementing data workflows using Apache NiFi for data ingestion, transformation, and distribution. - Collaborating with cross-functional teams to understand data requirements and deliver efficient solutions. - Ensuring data quality, governance, and security standards are met within the on-premise infrastruc...

As a DevOps Engineer, you will be responsible for managing, designing highly scalable and available solutions for data pipelines that provide the foundation for collecting, storing, modelling, and analyzing massive data sets from multiple channels. Experience: 4 to 7 Years Location: Bangalore Notice period: 0 to 45 days Qualifications: 1) 4 - 7 years track record of relevant work experience and a computer Science or a related technical discipline is required 2) Proven track record of building and shipping large-scale engineering products and/or knowledge of cloud infrastructure such as Azure/AWS preferred 3) Experience with Python or Java programming. 4) Experience in Shell, or any scripting...

Job description Key Responsibilities: Design, build, and optimize scalable ETL/ELT pipelines using PySpark on both Hadoop and GCP environments. Utilize Hadoop ecosystem components including Hive , HDFS , YARN , and Oozie for data storage, processing, and workflow orchestration. Implement and manage GCP data services such as BigQuery , Cloud Storage , Dataflow , and Cloud Composer to support data ingestion, transformation, and analytics workloads. Ensure the performance, scalability, and reliability of data pipelines to meet business and operational requirements. Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data needs and...

Role & responsibilities 1. Strong experience in Bigdata development 2. Experience in Python and Pyspark 3. Experience in Unix/Shell scripting 4. Experience in Hadoop,Hive,Scoop,Scala

As a senior Applications Development Technology Lead Analyst at the company, your primary role is to establish and implement new or revised application systems and programs in coordination with the Technology team. Your objective will be to lead applications systems analysis and programming activities. **Key Responsibilities:** - Translate application storyboards and use cases into functional applications. - Design, build, and maintain efficient, reusable, and reliable Java code. - Ensure the best possible performance, quality, and responsiveness of the applications. - Identify bottlenecks and bugs, and devise solutions to these problems. - Develop high performance & low latency components t...

As a Senior Engineer at Impetus Technologies, you will be responsible for designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your role will involve collaborating with cross-functional teams to gather requirements, develop high-quality code, and optimize data processing workflows. Additionally, you will mentor junior engineers and contribute to architectural decisions to enhance system performance and scalability. Key Responsibilities: - Design, develop, and maintain high-performance applications using Java and Big Data technologies. - Implement data ingestion and processing workflows utilizing frameworks like Hadoop and Spark. - C...

Your IT Future, Delivered. Principal DevOps Engineer With a global team of 5600+ IT professionals, DHL IT Services connects people and keeps the global economy running by continuously innovating and creating sustainable digital solutions. We work beyond global borders and push boundaries across all dimensions of logistics. You can leave your mark shaping the technology backbone of the biggest logistics company of the world. All our locations have earned the #GreatPlaceToWork certification, reflecting our commitment to exceptional employee experience. Digitalization. Simply delivered. At DHL IT Services, we are designing, building and running IT solutions for the whole DPDHL globally. The AI ...

The Data Engineer will serve as a technical expert in the fields ofdesign and develop AI data pipelines to manage both large unstructuredand structured datasets, with a focus on building data pipelines for enterprise AI solutions Job Description In your new role you will: Working closely with data scientists and domain experts to design anddevelop AI data pipelines using agile development process. Developing pipelines for ingesting and processing large unstructuredand structured datasets from a variety of sources, with a specificemphasis on creating solutions for AI solutions to ensure efficient andeffective data processing. Work efficiently with structured and unstructured data sources. Wor...

Req ID: 338133 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Cloudera Solution Specialist to join our team in Bangalore, Karn?taka (IN-KA), India (IN). Job Duties: Key Responsibilities: Develop and articulate long-term strategic goals for data architecture vision and establish data standards for enterprise systems. Utilize various cloud technologies, including Azure, AWS, GCP, and data platforms like Databricks and Snowflake. Conceptualize and create an end-to-end vision outlining the seamless flow of data through ...

Job description Some careers shine brighter than others. If you're looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further. HSBC is one of the largest banking and financial services organisations in the world, with operations in 64 countries and territories. We aim to be where the growth is, enabling businesses to thrive and economies to prosper, and, ultimately, helping people to fulfil their hopes and realise their ambitions. We are currently seeking an experienced professional to...

Position Overview Job Title - GCP - Senior Engineer - PD Location - Pune Role Description Our team is part of the area Technology, Data, and Innovation (TDI) Private Bank. Within TDI, Partner data is the central client reference data system in Germany. As a core banking system, many banking processes and applications are integrated and communicate via >2k interfaces. From a technical perspective, we focus on mainframe but also build solutions on premise cloud, restful services, and an angular frontend. Next to the maintenance and the implementation of new CTB requirements, the content focus also lies on the regulatory and tax topics surrounding a partner/ client. We are looking for a very mo...

Role Overview: As an Applications Development Technology Lead Analyst at our company, you will be a senior member responsible for establishing and implementing new or revised application systems and programs in coordination with the Technology team. Your main objective in this role will be to lead applications systems analysis and programming activities. You should have at least two years of experience in building and leading highly complex, technical data engineering teams, with an overall of 10+ years of hands-on Data Engineering experience. Key Responsibilities: - Partner with multiple management teams to ensure appropriate integration of functions and identify necessary system enhancemen...

Times Internet is the largest digital products company and the digital venture of Times of India, India's largest media and entertainment group. TIL websites are among the fastest growing Web / Mobile based networks worldwide. Since its inception in 1999, Times Internet has led the Internet revolution in India and has emerged as India's foremost web entity, running diverse portals and niche websites. **Role Overview:** As a MySql Database Admin at Times Internet in NOIDA, you will participate in the design, implementation, automation, optimization, and ongoing operational administration related tasks backend system running on MySQL/MariaDB/ MongoDB/RedisDB/Hadoop database infrastructure. You...

C1X AdTech Pvt Ltd is a fast-growing, product and engineering-driven AdTech company building next-generation advertising and marketing technology platforms. Our mission is to empower enterprise clients with the smartest marketing solutions, enabling seamless integration with personalization engines and delivering cross-channel marketing capabilities. We are dedicated to enhancing customer engagement and experiences while focusing on increasing Lifetime Value (LTV) through consistent messaging across all channels. Our engineering team spans front end (UI), back end (Java/Node.js APIs), Big Data, and DevOps, working together to deliver scalable, high-performance products for the digital advert...

Let's be unstoppable together! At Circana, we are fueled by our passion for continuous learning and growth, we seek and share feedback freely, and we celebrate victories both big and small in an environment that is flexible and accommodating to our work and personal lives. We're a global company dedicated to fostering inclusivity and belonging. We value and celebrate the unique experiences, cultures, and viewpoints that each individual brings. By embracing a wide range of backgrounds, skills, expertise, and beyond, we create a stronger, more innovative environment for our employees, clients, and communities. Circana is proud to be Certified by Great Place To Work. This prestigious award is b...

As a Data Engineer at Valorem Reply, you will be responsible for designing and contributing to the architecture of database systems, including both normalized transactional systems and dimensional reporting systems. Your expertise with SQL Server and Microsoft technologies, as well as experience in implementing Big Data solutions using Databricks, will be essential for success in this role. You will play a key role in providing data solutions to support business applications and reporting systems, working closely with customers to deliver effective solutions. Proficiency in Power BI and Azure Data Services is a must to excel in this position. Key Responsibilities: - Design and implement arch...

Job description Some careers shine brighter than others. If you're looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further. HSBC is one of the largest banking and financial services organizations in the world, with operations in 64 countries and territories. We aim to be where the growth is, enabling businesses to thrive and economies to prosper, and, ultimately, helping people to fulfil their hopes and realize their ambitions. We are currently seeking an experienced professional to...

Role Overview: You will be joining the ICG TTS Operations Technology (OpsTech) Group to assist in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role as a Lead Spark Scala Engineer will involve managing development teams in the distributed systems Eco-System and collaborating effectively as a team player. Your primary objective will be to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical leadership and guidance to a team of ...

As a Senior Data Engineer (Scala) at EY in the Advisory - Data and Analytics team, you will play a crucial role in leveraging your expertise in the Financial Services domain to solve complex issues and capitalize on opportunities that deliver better outcomes for businesses. Your primary responsibilities will include: - Demonstrating strong knowledge in Spark, with a good understanding of the Spark framework and performance tuning. - Proficiency in Scala & SQL to effectively work with data. - Exposure to at least one of the Cloud technologies - GCP/ Azure/ AWS. - Designing, building, and maintaining scalable data pipelines and solutions to efficiently manage and process large datasets. - Taki...

Role Overview As a Big Data Engineer, you will be responsible for building and maintaining scalable data pipelines and architectures that support data analytics, reporting, and real-time processing. You will collaborate with data scientists, business analysts, and software engineers to design robust solutions that drive data-driven decisions. Mandatory Skill Set Extensive hands-on experience with the Hadoop ecosystem (HDFS, MapReduce, YARN). Strong expertise in Apache Spark for large-scale data processing and analytics. Proficiency in Hive writing complex HiveQL queries and managing Hive Metastore. Experience in developing and maintaining Apache Kafka-based streaming solutions. In-depth know...

Role - Data Engineer Experience - 3+ yrs Location - Bangalore Roles and Responsibilities: Create and maintain optimal data pipeline architecture Create and maintain events/streaming based architecture/design Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies. Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative indust...

Data Engineer should have extensive knowledge in different programming or scripting languages [Python] Good understanding in streaming (Kafka/Storm/Kinesis) Good understanding on Big data components (HDFS, YARN, Map Reduce, Spark, Oozie) Good understanding on Azure components (ADF, ADB, ADLS) Good understanding Version controlling (Git, GitHub, azure DevOps). Competency in Cloud environment is must [preferred, Azure] Should have experience in transforming raw/unstructured data to clean data [Data Quality] SKILLS DESIRED (Technical & Behavioural) Secondary Microsoft Certified: Azure Data Engineer Associate Knowledge of HDFS, ADLS and S3(Simple Storage service). Query tuning, server optimizati...