896 Hdfs Jobs - Page 15

Key skills: Spark, Scala Experience: 6 to 12 years Location: AIA Hyderabad Job description Develop, test, and deploy data processing applications using Apache Spark and Scala. Optimize and tune Spark applications for better performance on large-scale data sets. Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. Design and implement high-performance data processing and analytics solutions. Ensure data integrity, accuracy, and security across all processing tasks. Troubleshoot and resolve pe...

Key activities Mandatory Skills: Location: Bangalore/Hyderabad Databricks: Expertise in design jobs and workloads. Proficiency in pyspark / scala for data manipulation, transformation and analysis o Optimization of Databricks clusters and notebooks. Python: Hands-on experience in automation and scripting. Cloudera: Strong knowledge of Spark, Hive, Impala, HDFS, Kafa, HBase and Cluster management. Palantir: Hands on experience with analytics, data fusion and developing analytical workflows DevOps Basics: Familiarity with Jenkins CI/CD pipelines. Integrate Databricks with AWS ecosystem components such as S3, Glue, Athena, Redshift, and Lambda Communication: Excellent verbal and written communi...

Essential Responsibilities: Desired Characteristics Minimum 7 years of software engineering experience. Proven long term experience and enthusiasm for distributed data processing at scale, eagerness to learn new things. Expertise in designing and architecting distributed low latency and scalable solutions in either cloud and on premises environment. Exposure to the whole software development lifecycle from inception to production and monitoring Fluency in Python or solid experience in Scala, Java Proficient with relational databases and Advanced SQL Expert in usage of services like Spark, HDFS, Hive, HBase Experience in adequate usage of any scheduler such as Apache Airflow, Apache Luigi, Ch...

Role Overview: As a test engineer at Arista Networks, you will be responsible for testing and validating complex cloud infrastructure and services that power WiFi networks. You will play a crucial role in ensuring that the WiFi networks perform at scale and meet the highest standards of reliability, security, and efficiency. Working closely with cross-functional teams, you will contribute to the design, development, and execution of comprehensive test plans for cloud-based WiFi solutions, ultimately delivering industry-leading products. Key Responsibilities: - Lead the testing efforts on the cloud infrastructure supporting WiFi networks, ensuring scalability, performance, and resilience in v...

As a Software Engineer at our dynamic team, you will be working with big data technologies such as Hadoop, Hive, and Spark to process and analyze large datasets. Your responsibilities will include collaborating with cross-functional teams to gather requirements and implement solutions that meet business needs. You will also be involved in all phases of the software development lifecycle, including planning, development, testing, and deployment. Additionally, you will be responsible for implementing CI/CD pipelines using tools like Jenkins to automate build, test, and deployment processes. Your ability to adapt quickly to new technologies and methodologies, troubleshoot and resolve issues in ...

As an experienced professional with over 7+ years of experience in Noida, your role will involve the following key responsibilities: - Reviewing and understanding business requirements to ensure timely completion of development tasks with minimal defects - Collaborating with a software development team to implement best practices and enhance the performance of Data applications for meeting client needs - Engaging with various teams and customers to translate their business challenges into innovative solutions - Conducting research on new Big Data technologies to align them with the business and technology strategy - Operating within a rapid and agile development process to accelerate speed t...

JD:- Total Yrs. of Experience* 6+ Relevant Yrs. of experience* 6+ Detailed JD *(Roles and Responsibilities) 6+ years experience in Kafka, Spark, Scala ,Pyspark Strong knowledge of implementing and maintaining scalable streaming and batch data solutions with high throughput and strict SLA. Extensive experience of working in production with Kafka, Spark and Kafka Stream including best practices, observability, optimization, performance tuning. Knowledge of Python and Scala including best practices, architecture, dependency management Collaborate with cross-functional teams to understand data requirements and integrate data from multiple sources, ensuring data consistency and quality Mandatory ...

Scala & Spark - Intermediate/Advanced level is a mandate. Data structures and algorithms in Scala. Minimum 4+ years of experience in Scala/Spark. 4. Need to have a big-data background

Data Engineer should have extensive knowledge in different programming or scripting languages [Python] Good understanding in streaming (Kafka/Storm/Kinesis) Good understanding on Big data components (HDFS, YARN, Map Reduce, Spark, Oozie) Good understanding on Azure components (ADF, ADB, ADLS) Good understanding Version controlling (Git, GitHub, azure DevOps). Competency in Cloud environment is must [preferred, Azure] Should have experience in transforming raw/unstructured data to clean data [Data Quality] SKILLS DESIRED (Technical & Behavioural) Secondary Microsoft Certified: Azure Data Engineer Associate Knowledge of HDFS, ADLS and S3(Simple Storage service). Query tuning, server optimizati...

What youll do: We seek Software Engineers with experience building and scaling services in on-premises and cloud environments. As a Principal Software Engineer in the Epsilon Attribution/Forecasting Product Development team, you will design, implement, and optimize data processing solutions using Scala, Spark, and Hadoop. Collaborate with cross-functional teams to deploy big data solutions on our on-premises and cloud infrastructure along with building, scheduling and maintaining workflows. Perform data integration and transformation, troubleshoot issues, Document processes, communicate technical concepts clearly, and continuously enhance our attribution engine/forecasting engine. Strong wri...

Role Description We are looking for a Senior Data Warehouse Engineer with good designing skills to work with Conversant Engineering Data Warehousing teams which deals in data of petabyte scale. This role will help to define solutions for given product and engineering initiatives , and drive solutions to delivery by effectively engaging with team members across the globe. The person in this role will be working closely with key stakeholders throughout the organization , so he or she must be able to communicate and keep them informed on overall health on projects impacting Conversant Data Warehouse platforms. He should also be able to mentor Juniors in the Teams. What you ll need Lead , design...

Role Description We are looking for a Senior Data Warehouse Engineer with good designing skills to work with Conversant Engineering Data Warehousing teams which deals in data of petabyte scale. This role will help to define solutions for given product and engineering initiatives , and drive solutions to delivery by effectively engaging with team members across the globe. The person in this role will be working closely with key stakeholders throughout the organization , so he or she must be able to communicate and keep them informed on overall health on projects impacting Conversant Data Warehouse platforms. He should also be able to mentor Juniors in the Teams. What you ll need Lead , design...

Role Description We are looking for a Senior Data Warehouse Engineer with good designing skills to work with Conversant Engineering Data Warehousing teams which deals in data of petabyte scale. This role will help to define solutions for given product and engineering initiatives , and drive solutions to delivery by effectively engaging with team members across the globe. The person in this role will be working closely with key stakeholders throughout the organization , so he or she must be able to communicate and keep them informed on overall health on projects impacting Conversant Data Warehouse platforms. He should also be able to mentor Juniors in the Teams. What you ll need Lead , design...

This position isresponsible for hands-on design&implementation expertiseinSpark and Python (PySpark) along with other Hadoop ecosystems like HDFS, Hive, Hue, Impala, Zeppelinetc. The purpose of positionincludes- Analysis, design and implementation of business requirements using SPARK& Python. Cloudera Hadoopdevelopment around Big Data. Solid SQL experience. Development experience with PySpark&SparkSql with good analytical & debugging skills. Development work for building new solutions around Hadoop and automation of operational tasks. Assisting team and troubleshooting issues. Required Skills PYTHON , HADOOP , SQL , SHELL SCRIPTING, Spark SQL

Data Engineer should have extensive knowledge in different programming or scripting languages [Python] Good understanding in streaming (Kafka/Storm/Kinesis) Good understanding on Big data components (HDFS, YARN, Map Reduce, Spark, Oozie) Good understanding on Azure components (ADF, ADB, ADLS) Good understanding Version controlling (Git, GitHub, azure DevOps). Competency in Cloud environment is must [preferred, Azure] Should have experience in transforming raw/unstructured data to clean data [Data Quality] SKILLS DESIRED (Technical & Behavioural) Secondary Microsoft Certified: Azure Data Engineer Associate Knowledge of HDFS, ADLS and S3(Simple Storage service). Query tuning, server optimizati...

Role Description We are looking for a Senior Data Warehouse Engineer with good designing skills to work with Conversant Engineering Data Warehousing teams which deals in data of petabyte scale. This role will help to define solutions for given product and engineering initiatives , and drive solutions to delivery by effectively engaging with team members across the globe. The person in this role will be working closely with key stakeholders throughout the organization , so he or she must be able to communicate and keep them informed on overall health on projects impacting Conversant Data Warehouse platforms. He should also be able to mentor Juniors in the Teams. What you ll need Lead , design...

Hi, We have positions for Data Scientist role - Airoli - WFO (5 DAYS) JD: Key Responsibilities: Develop and maintain data pipelines using Hadoop ecosystem. Build and maintain Spark SQL applications for big data processing and analysis. Use Tableau to visualize data, create dashboards, and extract insights from Hadoop data. Optimize data processing performance, including query optimization and data storage strategies. Collaborate with other engineers and stakeholders to understand data requirements and deliver solutions. Troubleshoot and resolve issues related to data processing and visualization. Stay up to date with the latest technologies and best practices in the field. A Hadoop Administr...

Role & responsibilities Overall, 6 to 8 years of experience in IT Industry. Min 6 years of experience working on Data Engineering. Design and develop solution in combination of Cloudera, Palantir and Databricks. Develop Python-based automation scripts to optimize workflows and processes. Integrate Databricks with AWS ecosystem components such as S3, Glue, Athena, Redshift, and Lambda Define architecture for data-driven solutions. Provide solutioning for web applications and data pipelines. Solutioning data analytics requirements with Palantir. Stay updated with emerging technologies and quickly adapt to new tools and frameworks. Work with CI/CD pipelines. Ensure scalability, security, and pe...

Role Overview: As a seasoned Senior Test Automation Engineer at Treasury and FP&A Technology, you will be responsible for defining, planning, and executing the testing automation strategy for the Global Funds Transfer Pricing Application. Your role will involve architecting robust automation frameworks, enhancing testing efficiency, and ensuring seamless software delivery of the highest quality. Your expertise in automation tools, agile methodologies, and quality engineering best practices will be crucial in transforming and enhancing the current testing automation landscape. You must have an engineering-first mindset, leveraging the latest technologies to achieve efficient and effective sol...

Spark, Scala, Bigdata Team Management: Lead and manage a team of technical professionals Data Engineer and Application Developers, ensuring effective collaboration and productivity, Client Interaction: Serve as the primary point of contact for clients, understanding their needs and ensuring successful project delivery, Status Reporting: Prepare and present detailed status reports on ongoing projects to stakeholders, highlighting progress, risks, and mitigation plans, Issue Resolution: Proactively identify and resolve any technical challenges or roadblocks faced by the team, ensuring smooth project execution, Project Management: Oversee the entire project lifecycle, from planning to execution...

Highly proficient in Databricks and Pyspark Strong python programming skills. Strong in SQL scripting. Good to have experiece in snowflake cloud datawarehouse Experience in AWS services like s3, cloudwatch, kms Required good debugging skills Nice to have exposure to airflow orchestration and CI/CD Good communication and problem solving skills

Job Description: We are looking for a Hadoop resource for running a Hadoop Assessment with the help of internal Hadoop Decoded tool. Hadoop Decoded tool analyses different logs including Hadoop, Yarn etc. to get insight into the cluster usage of Hadoop ecosystem. The tool parses and ingests logs into Elasticsearch via Logstash, and visualizations are built in Kibana. Key Responsibilities - Understand and enhance (if required) the existing Hadoop Decoded codebase and pipeline architecture. Manage data ingestion workflows using Logstash and indexing in Elasticsearch. Optimize Kibana dashboards for visualization and reporting as required Execute Hadoop Decoded on the Hadoop cluster and deliver ...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Principal Apache Hadoop Engineer at BairesDev We are seeking a Principal Apache Hadoop Engineer with deep expertise in big data ecosystem, HDFS ...

Job Title: - GCP Data Engineer with Big Query ( AIA Kolkata ) Location: - AIA Kochi Minimum 6 to 9 years of experience Mandatory Skills: GCP Services – Big Query, Cloud SQL, Data flow Around 7+ years of experience in development in Data warehousing projects in GCP platforms, Good understanding of dimensional modelling Expertise in MySQL & SQL/PL GCP & BigQuery knowledge is must, GCP certification is added advantage Good experience in Cloud Composer, DAGs , airflow REST API development experience Good in analytical and problem solving, efficient communication Ability to lead the team members, convert requirements into technical solutions, assign the work, coordinate the team and review the de...

Loc: Manyata Tech Park Exp: 2-5 yrs, AI/ML Engineer Skills Required: ? Exp. with Data engineering & Data/ML pipelines for Analytics & AI/ML Models ? Working with large sets of structured & unstructured data from disparate sources ? Foundational acquaintance of Data Science & AI/ML models ? SQL, Python, Spark, PySpark ? Big Data systems - Hadoop Ecosystem (HDFS, Hive, MapReduce) ? Analytics database like Druid, Data visualization/exploration tools like Superset ? Understanding of cloud platforms (AWS, Azure, or GCP) ? GIT Secondary Skills (desired/preferred): ? Apache Airflow, Apache Oozie, Nifi ? GCP cloud exp, Big Query, GCP Big Data Ecosystem ? Trino/Presto ? Familiar with forecasting algo...