895 Hdfs Jobs - Page 5

Technical Expertise Solid Python programming (OOPS, REST, API), SQL & Linux experience, handling ODBC/JDBC/Arrowflight connections. Have done projects handling large volume of data, implemented MMP tools and strong knowledge of storage solutions (NAS, S3, HDFS) and data formats (parquet, avro, iceberg) Strong knowledge of Kubernetes, containerization, Helm chart, templates and overlay, Vault, SSL/TSL, KB stores Prior knowledge of key libraries e.g. S3, MongoDB, Elastic, Trident NAS, Grafana etc will be big plus. Used Git and built CI-CD pipelines in recent projects. Additional Criteria Proactive in asking questions, indulge with team in group conversations, actively participate in issues dis...

Must have: 7+ years of development experience in Java Good background with Spring framework Project experience(s) on Java Spark Job using HDFS Strong SQL backgroundNice to have: Snowflake Airflow, Angular, NodeJS, Tableau Mandatory Competencies Programming Language - Java - Core Java (java 8+) Programming Language - Java - Spring Framework Beh - Communication Database - Database Programming - SQL Big Data - Big Data - SPARK

As a Senior Data and AI Engineer at TBO, you will play a crucial role in designing, developing, and maintaining data pipelines and ETL workflows using Apache Spark and AWS Glue. Your responsibilities will include building and optimizing large-scale data lake solutions leveraging HDFS, Apache Hudi, and AWS S3. Additionally, you will be involved in developing and managing data ingestion frameworks, transformations, and data modeling for analytical workloads. You will also design and develop AI/ML pipelines using pyspark, LLM, Langchain, RAG to solve business usecases. Furthermore, you will enable data discovery and querying using AWS Athena and other analytical tools, and implement and monitor...

As a Hadoop Engineer at our company, your role involves managing Hadoop clusters and performing core administrative tasks to ensure the smooth functioning of the Big Data ecosystem. Your practical experience in Hadoop technologies will be essential for the following key responsibilities: - Install, configure, and maintain Hadoop clusters along with ecosystem components like HDFS, YARN, Hive, Spark, etc. - Monitor clusters, troubleshoot issues, and optimize performance to guarantee reliability and high availability. - Conduct performance tuning, capacity planning, and load balancing across Hadoop nodes. - Manage data backup, recovery procedures, and implement necessary security configurations...

As a Senior Data Engineer Privacy at Roku, your primary responsibility will be to ensure compliance with privacy regulations by designing, implementing, and maintaining robust data privacy measures within the organization. You will collaborate with cross-functional teams to conduct risk assessments, develop strategies to safeguard sensitive information, and enhance data governance frameworks. Your role will involve working on Rokus Big Data Platform architecture design, data modelling, and ETL to ensure compliance with privacy requirements. You will also be responsible for owning requirements, development, and quality assurance of Privacy compliance tools within Rokus distributed data proces...

Role Overview: You are a seasoned IT Quality Sr Analyst who will be responsible for contributing to the development of new techniques and enhancing processes for the Treasury and FP&A Technology team. You will be required to integrate industry expertise within the defined area and evaluate complex issues with substantial impact. Your strong analytical skills and communication abilities will be crucial in assuming leadership roles within teams and influencing decisions through advice and counsel. Key Responsibilities: - Plan, lead, and execute testing automation strategy for CitiFTP, monitoring automation coverage and enhancing existing frameworks. - Design, develop, and implement scalable au...

As an experienced Big Data Engineer, you will be responsible for the following: - Designing, developing, and optimizing scalable Big Data pipelines. - Working closely with cross-functional teams on data acquisition, transformation, and processing. - Performing ETL workflows, data ingestion, and data processing using the Hadoop ecosystem. - Building and maintaining data solutions ensuring performance, scalability, and reliability. - Monitoring, troubleshooting, and tuning data pipelines to ensure optimal performance. Mandatory Skills: - Big Data / Hadoop Technologies: Hive, HQL, HDFS - Programming: Python, PySpark, SQL (Strong query writing) - Schedulers: Control-M or equivalent scheduler - D...

Role Value Proposition: Work and collaborate with a nimble, autonomous, cross-functional team of makers, breakers, doers, and disruptors who love to solve real problems and meet real customer needs. You will be using cutting-edge technologies and frameworks to analyze data, to help create the pipeline, and collaborate with the data science team to enable the innovative work in machine learning and AI. Eagerness to learn new technologies on the fly and ship to production Knowledge in data science is a plus More than just a job we hire people who love what they do! Job Responsibilities 1. Building and Implementing data ingestion and curation process developed using Big data tools such as Spark...

Your role and responsibilities As an Application Developer, you will lead IBM into the future by translating system requirements into the design and development of customized systems in an agile environment. The success of IBM is in your hands as you transform vital business needs into code and drive innovation. Your work will power IBM and its clients globally, collaborating and integrating code into enterprise systems. You will have access to the latest education, tools and technology, and a limitless career path with the worlds technology leader. Come to IBM and make a global impact Responsibilities: Responsible to manage end to end feature development and resolve challenges faced in impl...

Your role and responsibilities As an Application Developer, you will lead IBM into the future by translating system requirements into the design and development of customized systems in an agile environment. The success of IBM is in your hands as you transform vital business needs into code and drive innovation. Your work will power IBM and its clients globally, collaborating and integrating code into enterprise systems. You will have access to the latest education, tools and technology, and a limitless career path with the worlds technology leader. Come to IBM and make a global impact Responsibilities: Responsible to manage end to end feature development and resolve challenges faced in impl...

Key skills: Informatica BDM/DEI, Hadoop/HDFS Experience level: 5+ years Evaluation criteria: Mandatory Requirements •Proven development experience with Informatica DEI tools (PowerCenter or BDM) •Strong understanding of Informatica BDM architecture and components •Hands-on experience with Hadoop/HDFS ecosystem: oHive SQL oFile system types and structures Preferred Skills (Good to Have) •Knowledge on Teradata/Cloudera •Agility in responding to tasks and challenges •Effective communication skills •Proactive and solution-oriented approach

Role & responsibilities Understanding & familiarity with all Hadoop Ecosystem components and Hadoop Administrative Fundamentals Experience working with NoSQL in at least one of the data stores - HBase, Cassandra, MongoDB Experience in HDFS, Hive, Impala Experience is schedulers like Airflow, Nifi etc. Experienced in Hadoop clustering and Auto scaling. Develop standardized practices for delivering new products and capabilities using Big Data technologies, including data acquisition, transformation, and analysis. Define and develop client specific best practices around data management within a Hadoop environment on Azure cloud Monitoring health, tuning, and growth of such on-premises and cloud...

5 to 10 Years of in Big Data & Data related technology experience Expert level understanding of distributed computing principles Expert level knowledge and experience in Apache Spark Hands on programing with Python Java Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming Experience with messaging systems, such as Kafka or RabbitMQ Good understanding of Big Data querying tools, such as Hive, and Impala Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files Good understanding of SQL queries, joins, stored procedures, relational sc...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Apache Ozone Developer at BairesDev We are seeking an Apache Ozone Developer with expertise in distributed storage systems, object storage archi...

Power BI Report Developer Excellent understanding on data architecture system (source, target, transformations, processing, etc.,) and migration b/w DB platforms. Ability to connect with various data source like SQL Server, Oracle, Azure SQL, SSAS, Azure Analysis Service, HDFS etc from Power BI Data Preparation using the inbuilt features of Power BI Experience in Data Modelling and custom calculations in Power BI Strong Experience in using complex DAX expressions Design reports with the inbuilt charts and customization of the charts Ability to use various filter options, Should be able to create Generic Reports, complex visuals and KPI charts Scheduling Data refresh in report server and User...

Key Responsibilities: Install, configure, and manage Hadoop clusters, including HDFS, YARN, Hive, HBase, and other ecosystem components. Monitor and manage Hadoop cluster performance, capacity, and security. Perform routine maintenance tasks such as upgrades, patching, and backups. Implement and maintain data ingestion processes using tools like Sqoop, Flume, and Kafka. Ensure high availability and disaster recovery of Hadoop clusters. Collaborate with development teams to understand requirements and provide appropriate Hadoop solutions. Troubleshoot and resolve issues related to the Hadoop ecosystem. Maintain documentation of Hadoop environment configurations, processes, and procedures. Req...

As an experienced Big Data Engineer, your role will involve the following responsibilities: - Designing, developing, and optimizing scalable Big Data pipelines. - Working closely with cross-functional teams on data acquisition, transformation, and processing. - Performing ETL workflows, data ingestion, and data processing using the Hadoop ecosystem. - Building and maintaining data solutions to ensure performance, scalability, and reliability. - Monitoring, troubleshooting, and tuning data pipelines to ensure optimal performance. Mandatory Skills: - Strong proficiency in Big Data / Hadoop Technologies such as Hive, HQL, and HDFS. - Proficient in programming languages like Python, PySpark, and...

As an experienced Big Data Engineer, your role will involve designing, developing, and optimizing scalable Big Data pipelines. You will work closely with cross-functional teams on data acquisition, transformation, and processing. Your key responsibilities will include performing ETL workflows, data ingestion, and data processing using the Hadoop ecosystem. It will be your responsibility to build and maintain data solutions ensuring performance, scalability, and reliability. Additionally, you will need to monitor, troubleshoot, and tune data pipelines to ensure optimal performance. Key Responsibilities: - Design, develop, and optimize scalable Big Data pipelines. - Work closely with cross-fun...

Role Overview: As a Big Data Engineer specializing in Spark & Scala, you will be responsible for developing and optimizing big data processing pipelines using Apache Spark and Scala. Your role will involve designing and implementing ETL workflows for large-scale batch and real-time data processing. Additionally, you will be expected to optimize Spark performance through partitioning, caching, and memory/shuffle tuning. Collaboration with cross-functional teams and adherence to best practices in coding, testing, and deployment are essential aspects of this role. Key Responsibilities: - Develop and optimize big data processing pipelines using Apache Spark (Core, SQL, Streaming) and Scala. - De...

Role Overview: You will be a Senior Engineer at Impetus Technologies, utilizing your expertise in Java and Big Data technologies to design, develop, and deploy scalable data processing applications. Your responsibilities will include collaborating with cross-functional teams, mentoring junior engineers, and contributing to architectural decisions to enhance system performance and scalability. Key Responsibilities: - Design, develop, and maintain high-performance applications using Java and Big Data technologies. - Implement data ingestion and processing workflows with frameworks like Hadoop and Spark. - Collaborate with the data architecture team to define efficient data models. - Optimize e...

Job Description: 5 to 10 Years of in Big Data & Data related technology experience Expert level understanding of distributed computing principles Expert level knowledge and experience in Apache Spark Hands on programing with Python Java Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming Experience with messaging systems, such as Kafka or RabbitMQ Good understanding of Big Data querying tools, such as Hive, and Impala Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files Good understanding of SQL queries, joins, stored procedur...

Qualification : Bachelor's/Master's Degree. Relevant Power BI Experience : 5+ Years (Report Development) Relevant DWH/ETL Experience : 5+ Years Overall IT Experience : 5 to 10 Years Power BI Report Developer Excellent understanding on data architecture system (source, target, transformations, processing, etc.,) and migration b/w DB platforms. Ability to connect with various data source like SQL Server, Oracle, Azure SQL, SSAS, Azure Analysis Service, HDFS etc from Power BI Data Preparation using the inbuilt features of Power BI Experience in Data Modelling and custom calculations in Power BI Strong Experience in using complex DAX expressions Design reports with the inbuilt charts and customi...

Design, build and maintain real-time and batch data pipelines using Python, PySpark, and AWS services. Architect and manage cloud infrastructure with a focus on scalability, cost-efficiency, and performance (AWS Glue, Lambda, S3, Athena). Implement and optimize ETL/ELT workflows , ensuring data quality and reliability. Integrate data from diverse sources, including APIs (REST/GraphQL), into centralized data warehouses. Work with streaming technologies like Kafka or Amazon Kinesis to handle high-throughput, low-latency data ingestion. Lead data modeling efforts and contribute to the design of enterprise data warehouses and data lakes . Collaborate with product, engineering, and analytics team...

We have a skilled PySpark Developer with 6+ years of Big Data experience, expertise in PySpark, Hive, SparkSQL, Sqoop, and strong ETL/data warehousing knowledge. Hands-on with Hadoop/Cloudera and Unix scripting—ideal for end-to-end data pipeline work

Skill: Azure Data bricks Experience: 4 to 9 years Location: AIA Bangalore Job Summary We are seeking a highly skilled Sr. Developer with 7 to 10 years of experience to join our dynamic team. The ideal candidate will have expertise in Databricks SQL Databricks Delta Lake Databricks Workflows and PySpark. This role involves working in a hybrid model with day shifts focusing on developing and optimizing data solutions to drive business success. Secondary Skill - SQL and ETL development Responsibilities Develop and implement data solutions using Databricks SQL to enhance data processing capabilities. Optimize data storage and retrieval processes with Databricks Delta Lake for improved performanc...