886 Hdfs Jobs - Page 3

Job Title: Machine Learning Experience Range: 6+ Years Location: Delhi/ Gurgaon/ Noida Required Skill set: Machine Learning: Python (Advance level with complex algorithms) R (Advance level with complex algorithms) Visualization Monitoring with Prometheus and Grafana Tableau and Click View Container and Storage: Kubernetes Docker Working knowledge of Data Management Platforms: NiFi, Kafka, HDFS, Hive PostgreSQL NetApp storage IBM Spectrum Scale storage FreeIPA role based access control (RBAC)

Join phData, a dynamic and innovative leader in the modern data stack. We partner with major cloud data platforms like Snowflake, AWS, Azure, GCP, Fivetran, Pinecone, Glean and dbt to deliver cutting-edge services and solutions. We're committed to helping global enterprises overcome their toughest data challenges. phData is a remote-first global company with employees based in the United States, Latin America and India. We celebrate the culture of each of our team members and foster a community of technological curiosity, ownership and trust. Even though we're growing extremely fast, we maintain a casual, exciting work environment. We hire top performers and allow you the autonomy to deliver...

Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose to uplift everyone, everywhere by being the best way to pay and be paid. Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa. Job Description New Payment Flows (NPF) division's charter is to capture new sour...

Your role and responsibilities As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the pl...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Staff Apache Hadoop Engineer at BairesDev We are seeking a Staff Apache Hadoop Engineer with deep expertise in big data ecosystem, HDFS architec...

Role : Data Engineer ( HDFS / Spark/ Druid) Exp : 4 - 6 years Location : Navi Mumbai Data Engineers - (HDFS / Spark/ Druid) Cluster management Disk allocations Scaling ETL Monitoring Kafka

We are hiring a skilled Informatica BDM (Big Data Management) Developer to design, develop, and optimize ETL and data integration workflows on modern data platforms The ideal candidate will have hands-on experience in building and maintaining data pipelines using Informatica BDM, integrating data from multiple structured and unstructured sources into enterprise data lakes or warehouses The role involves working with Hadoop or cloud ecosystems (AWS, Azure, GCP) to process large-scale datasets, ensuring high performance, scalability, and reliability Candidates should possess strong ETL development, data warehousing, and performance tuning skills along with solid SQL knowledge Immediate joiners...

Must Have: ? SQL: Ability to read, write, test, and maintain complex SQL queries ? BigQuery: Working knowledge, including aspects like internal vs external tables, partitioning, clustering, etc. ? Python: Hands-on expertise in writing Python application code ? Big Data Engineering: Experience in Big Data Engineering projects Good to Have (Added Advantage) ? Knowledge/experience with DBT and Alembic (availability of these skills will be a big push) Experience Requirement: ? Mid-level engineer with 7 to 8 years of experience

Role Overview: As a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group, you will be responsible for developing and optimizing production-grade Spark applications in Scala. Your role will involve providing technical leadership, conducting code reviews, enforcing best practices, and optimizing performance. Your expertise in Spark, Scala, RDBMS, CI/CD tools, and other relevant technologies will be crucial in contributing to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical lea...

Role Overview: You will be responsible for designing and implementing general architecture for complex data systems, translating business requirements into functional and technical specifications, and developing and managing cloud-based data architecture and reporting solutions. Additionally, you will apply data modeling principles for relational and dimensional data structures, design Data Warehouses, and create and manage source-to-target mappings for ETL/ELT processes. Mentoring junior engineers and contributing to architectural decisions and code reviews will also be part of your key responsibilities. Key Responsibilities: - Design and implement general architecture for complex data syst...

Role Overview: You will be working as a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group to assist in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role will involve managing development teams in the distributed systems Eco-System and collaborating effectively as a team player. Your primary objective will be to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical leadership and guidance to a team o...

Job Description: As a Data Engineer at our company, you will be responsible for building and optimizing data pipelines using Apache Spark and SQL. You should have a minimum of 3 years of hands-on experience in this field. Your primary focus will be on batch data processing, Data Streaming, data reliability, and performance optimization. You will work closely with different teams to create scalable data solutions that align with the company's analytics and business goals. Key Responsibilities: - Utilize Apache Spark, including SparkSQL, for developing and optimizing data pipelines. - Demonstrate strong SQL skills by writing complex and optimized queries. - Implement batch processing technique...

As a Streamsets ETL Developer (AVP) at Deutsche Bank in Pune, India, you will play a crucial role in the Project Proteus, which aims to transform DWS into a standalone Asset Management platform. Your expertise will contribute to the strategic future state technology landscape for all DWS Corporate Functions globally. **Key Responsibilities:** - Collaborate with clients to deliver high-quality software within an agile development lifecycle. - Develop and rigorously test ETL solutions and pipelines. - Define and enhance the architecture of components, contributing to department and bank-wide architectural decisions. - Demonstrate end-to-end accountability for work/projects, following robust en...

Role Overview: As a Technical Support Engineer specializing in Big Data applications and systems within the Hadoop ecosystem, your primary responsibility is to provide technical support and troubleshooting to ensure the smooth functioning of these systems. You will collaborate with engineering teams, monitor system performance, and assist in the deployment, configuration, maintenance, and upgrades of Hadoop clusters. Your role also includes developing documentation, implementing data backup procedures, and staying updated with the latest Hadoop technologies and support methodologies. Additionally, you will participate in on-call rotations and contribute to the training and onboarding of new ...

Job Title AWS Data Senior Total IT Experience (in Yrs.) 7 to 11 ; Location - Pune Key words to search in resume AWS, glue, S3 , ETL Developer, Pyspark Developer Technical/Functional Skills -MUST HAVE SKILLS Knowledge of broad AWS Services, Experience in Backend development/maintenance preferably using PySpark/Python, Awareness of aws cost calculations. Troubleshooting abilities. Secondary Skills Data Management solutions with capabilities such as Metadata and Catalog, Performance and capacity Optimization, Data Security, Data Modeling, Data Wrangling, DevOps and basic agile awareness. Responsibilities Write aws glue/lambda jobs for sourcing data from multiple source systems to DataLake. Unit...

As a Senior Data Engineer at EY Advisory - Data and Analytics, you will have the opportunity to leverage your expertise in the Financial Services domain to design, build, and maintain scalable data pipelines and solutions. Your role will involve working with big data ecosystems and cloud technologies like GCP, Azure, or AWS to process large datasets efficiently. Here are the primary skills and key responsibilities for this role: - Strong knowledge in Spark and a good understanding of the Spark framework, including performance tuning. - Proficiency in Scala and SQL. - Hands-on experience in designing and building scalable data pipelines. - Experience with one of the Cloud technologies - GCP, ...

As a MySql Database Admin at Times Internet, you will play a crucial role in the design, implementation, automation, optimization, and ongoing operational administration of the backend system running on MySQL/MariaDB/MongoDB/RedisDB/Hadoop database infrastructure. You will be expected to take on instant support and operational challenges of infrastructure platforms running database services. Key Responsibilities: - Hands-on experience on MySQL/MariaDB RDBMS and related tools like Percona Xtrabackup and Percona-tool-kit - Hands-on experience on MongoDB NoSQL and Cache (RedisDB) datastores - Hands-on experience working on Private as well Public IT Infrastructure clouds like AWS - Hands-on expe...

Role Overview: You will be responsible for establishing and implementing new or revised application systems and programs in coordination with the Technology team. Your main objective will be to lead applications systems analysis and programming activities. Additionally, you will lead a data engineering team, from sourcing to closing, and drive the strategic vision for the team and product. Key Responsibilities: - Partner with multiple management teams to ensure appropriate integration of functions and identify necessary system enhancements - Manage a data-focused product and ML platform - Design, develop, and optimize scalable distributed data processing pipelines using Apache Spark and Scal...

Req ID: 345781 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Web-Java Architect to join our team in Bangalore, Karn?taka (IN-KA), India (IN). Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the ap...

Roles and Responsibilities Design, develop, test, deploy, and maintain large-scale data processing pipelines using Apache Spark. Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions. Troubleshoot complex issues related to HDFS, Hive, Pyspark, and other big data technologies. Ensure scalability, performance, and reliability of the system by implementing efficient algorithms and data structures. Participate in code reviews and contribute to the development of best practices for software engineering. Desired Candidate Profile 5-10 years of experience in working with Apache Spark (Scala/Python) on Hadoop ecosystem (HDFS). Strong understanding of Big D...

At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success. Staff Apache Ozone Engineer at BairesDev We are seeking a Staff Apache Ozone Engineer with deep expertise in distributed storage systems, object...

As a Technical Project Manager, you will be leading a team to build a highly-scalable and extensible big data platform that provides the foundation for collecting, storing, modeling, and analyzing massive data sets from multiple channels. Responsibilities : Align Sigmoid with key Client initiatives Interface daily with customers across leading Fortune 500 companies to understand strategic requirements Connect with CIO, VP and Director level clients on a regular basis. Travel to client locations Ability to understand business requirements and tie them to technology solutions Build a delivery plan with domain experts and stay on track Design, develop and evolve highly scalable and fault-tolera...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform * Responsibilities: * Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS * Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform * Experience in d...

Databricks, PySpark, Azure Data Factory, Python

Qualifications: The successful candidate should have either of the below B.SC, B.E/B.Tech./M.E./M.S./M.Tech with good academic record Experience: 10+ years overall with 4-5 years as Architect 4+ years of experience applying statistical/ML algorithms and techniques to real-world data sets. 4+ years as a developer with experience in related scalable and secure applications. 2+ years experience independently designing core product modules or complex components Hands on Experience on GenAI Key Responsibilities Involves in Designing and Overseeing the overall architecture of ML systems, including data pipelines, model selection, deployment strategies, and ensuring scalability and performance. Req...