141 Hdfs Jobs - Page 6

Candidate must be 4+ Year of development Experience using BigData tools/ecosystem like HDFS, Hive/HQL, Shell Scripting, PYSpark.Comfortable with writing straightforward SQL queries and advanced SQL queries using joins.Must have Knowledge of NoSQL databases/Hadoop-Hive, Spark etc. Good communication skillsGood in troubleshootingGood team player Qualification Candidate must be 4+ Year of development Experience using BigData tools/ecosystem like HDFS, Hive/HQL, Shell Scripting, PYSpark.Comfortable with writing straightforward SQL queries and advanced SQL queries using joins.Must have Knowledge of NoSQL databases/Hadoop-Hive, Spark etc. Good communication skillsGood in troubleshootingGood team player

Minimum 10 years experience in design, architecture or development in Analytics and Data Warehousing Have experience in solution design, solution governance and implementing end-to-end Big Data solutions using Hadoop eco-systems (Hive, HDFS, Pig, HBase, Flume, Kafka, Sqoop, YARN, Impala) Possess ability to produce semantic, conceptual, logical and physical data models using data modelling techniques such as Data Vault, Dimensional Modelling, 3NF, etc. Has the ability to design data warehousing and enterprise analytics-based solutions using Teradata or relevant data platforms Can demonstrate expertise in design patterns (FSLDM, IBM IFW DW) and data modelling frameworks including dimensional, star and non-dimensional schemas Possess commendable experience in consistently driving cost effective and technologically feasible solutions, while steering solution decisions across the group, to meet both operational and strategic goals is essential. Are adept with abilities to positively influence the adoption of new products, solutions and processes, to align with the existing Information Architectural design would be desirable Have Analytics & Data/BI Architecture appreciation and broad experience across all technology disciplines, including project management, IT strategy development and business process, information, application and business process. Have extensive experience with Teradata data warehouses and Big Data platforms on both On-Prim and Cloud platform. Extensive experience in large enterprise environments handling large volume of datasets with High Service Level Agreement(s) across various business functions/ units. Have experience leading discussions and presentations. Experience in driving decisions across groups of stakeholders.

PySpark, ANSI SQL, HDFS, Hive, Python, Scala, Java, Airflow, Any ETL tool, Shell Scripts, Control-M, Git, CI & CD, Unix File System, RDBMS Empirical understanding of data systems (BI analytics, data blending, data warehousing, data ETL, etc.) on process automation and standardization Good experience in requirement gathering/ analysis, coding, Unit testing, SIT Support Knowledge working on both Agile and waterfall models Develop end to end framework 4+ Years of relevant Experience in ETL and Data Engineering Strong Knowledge in Spark, Python Strong experience in Hive/SQL, PL/SQL Good Understanding of ETL & DW Concepts, Unix Scripting Design, implement and maintain Data Pipeline to meet the business requirements. Convert the Business need into Technical complex PySpark Code. Ability to write complex SQL queries for reporting purpose. Strong written and verbal communication, and proficiency in reporting analysis tools. Understanding of any ETL tool Understanding of any BI tool Cloud Computing AWS/AzureGood understanding of SDLC Core Compliance Domain knowledge Experience in the Agile Scrum Methodology

SUMMARY This is a remote position. Job Description: EMR Admin We are seeking an experienced EMR Admin with expertise in Big data services such as Hive, Metastore, H-base, and Hue. The ideal candidate should also possess knowledge in Terraform and Jenkins. Familiarity with Kerberos and Ansible tools would be an added advantage, although not mandatory. Additionally, candidates with Hadoop admin skills, proficiency in Terraform and Jenkins, and the ability to handle EMR Admin responsibilities are encouraged to apply. Location: Remote Experience: 6+ years Must-Have: The candidate should have 4 years in EMR Admin. Requirements Requirements: Proven experience in EMR administration Proficiency in Big data services including Hive, Metastore, H-base, and Hue Knowledge of Terraform and Jenkins Familiarity with Kerberos and Ansible tools (preferred) Experience in Hadoop administration (preferred)

We are looking for a highly skilled and motivated Data Engineer to join our dynamic team. In this role, you will collaborate with cross-functional teams to design, build, and maintain scalable data platforms on the AWS Cloud. Youll play a key role in developing next-generation data solutions and optimizing current implementations. Key Responsibilities: Build and maintain high-performance data pipelines using AWS Glue, EMR, Databricks, and Spark. Design and implement robust ETL processes to integrate and analyze large datasets. Develop and optimize data models for reporting, analytics, and machine learning workflows. Use Python, PySpark, and SQL for data transformation and optimization. Ensure data governance, security, and performance on AWS Cloud platforms. Collaborate with stakeholders to translate business needs into technical solutions. Required Skills & Experience: 3-5 years of hands-on experience in data engineering. Proficiency in Python, SQL, and PySpark. Strong knowledge of Big Data ecosystems (Hadoop, Hive, Sqoop, HDFS). Expertise in Spark (Spark Core, Spark Streaming, Spark SQL) and Databricks. Experience with AWS services like EMR, Glue, S3, EC2/EKS, and Lambda. Solid understanding of data modeling, warehousing, and ETL processes. Familiarity with data governance, quality, and security principles. Location - Anywhere in india,hyderabad,ahmedabad,pune,chennai,kolkata.

About the Company : Headquartered in California, U.S.A., GSPANN provides consulting and IT services to global clients. We help clients transform how they deliver business value by helping them optimize their IT capabilities, practices, and operations with our experience in retail, high-technology, and manufacturing. With five global delivery centers and 1900+ employees, we provide the intimacy of a boutique consultancy with the capabilities of a large IT services firm. Role: Big Data Developers/Lead Work Location: Bangalore (CV Raman Nagar), Pune, Hyderabad, Gurugram, Noida Experience : 5+ Years Technical Skill: BigData, AWS, redshift, snowflake, Spark, Python, Scala, & SQL Roles and Responsibilities 5+ Years of Big Data development experience with minimum 2 years hands-on in Java Hands-on experience of API development (from application / software engineering perspective) Advanced level experience (5+ years) building Real time streaming and batch systems using Apache Spark and Kafka in Java programming language. Experience with any NoSQL stores (HBase, Cassandra, MongoDB, Influx DB). Solid understanding of secure application development methodologies Experience in developing microservices using spring framework is a plus Capable of working as an individual contributor and within team too Design, build & maintain efficient, reusable & reliable code Experience in Hadoop based technologies Java, Hive, Pig, Map Reduce, Spark, python/Scala , Azure Should be able to understand complex architectures and be comfortable working with multiple teams Excellent communication, client engagement and client management skills are strongly preferred. Minimum Bachelors degree in Computer Science, Engineering, Business Information Systems, or related field. If the above profile suits you then request, to share your updated profile with below HR details: Full Name- Email Id- Phone No- Total years of experience - Relevant experience Bigdata- Relevant experience AWS- Relevant experience in Snowflake- Relevant experience in Redshift- Rating on SQL (out of5 )- Any other Technology- Notice period - CTC- ECTC- Current company- Current location: Preferred location: Any offers, If yes, Pls mention- Interview availability; Pls mention the Date and Time Revert with your confirmation

About the Company : Headquartered in California, U.S.A., GSPANN provides consulting and IT services to global clients. We help clients transform how they deliver business value by helping them optimize their IT capabilities, practices, and operations with our experience in retail, high-technology, and manufacturing. With five global delivery centers and 1900+ employees, we provide the intimacy of a boutique consultancy with the capabilities of a large IT services firm. Role: Big Data Developers/Lead Work Location: Bangalore (CV Raman Nagar), Pune, Hyderabad, Gurugram, Noida Experience : 5+ Years Technical Skill: BigData, AWS, redshift, snowflake, Spark, Python, Scala, & SQL Roles and Responsibilities 5+ Years of Big Data development experience with minimum 2 years hands-on in Java Hands-on experience of API development (from application / software engineering perspective) Advanced level experience (5+ years) building Real time streaming and batch systems using Apache Spark and Kafka in Java programming language. Experience with any NoSQL stores (HBase, Cassandra, MongoDB, Influx DB). Solid understanding of secure application development methodologies Experience in developing microservices using spring framework is a plus Capable of working as an individual contributor and within team too Design, build & maintain efficient, reusable & reliable code Experience in Hadoop based technologies Java, Hive, Pig, Map Reduce, Spark, python/Scala , Azure Should be able to understand complex architectures and be comfortable working with multiple teams Excellent communication, client engagement and client management skills are strongly preferred. Minimum Bachelors degree in Computer Science, Engineering, Business Information Systems, or related field. If the above profile suits you then request, to share your updated profile with below HR details: Full Name- Email Id- Phone No- Total years of experience - Relevant experience Bigdata- Relevant experience AWS- Relevant experience in Snowflake- Relevant experience in Redshift- Rating on SQL (out of5 )- Any other Technology- Notice period - CTC- ECTC- Current company- Current location: Preferred location: Any offers, If yes, Pls mention- Interview availability; Pls mention the Date and Time Revert with your confirmation

About the Company : Headquartered in California, U.S.A., GSPANN provides consulting and IT services to global clients. We help clients transform how they deliver business value by helping them optimize their IT capabilities, practices, and operations with our experience in retail, high-technology, and manufacturing. With five global delivery centers and 1900+ employees, we provide the intimacy of a boutique consultancy with the capabilities of a large IT services firm. Role: Big Data Developers/Lead Work Location: Bangalore (CV Raman Nagar), Pune, Hyderabad, Gurugram, Noida Experience : 5+ Years Technical Skill: BigData, AWS, redshift, snowflake, Spark, Python, Scala, & SQL Roles and Responsibilities 5+ Years of Big Data development experience with minimum 2 years hands-on in Java Hands-on experience of API development (from application / software engineering perspective) Advanced level experience (5+ years) building Real time streaming and batch systems using Apache Spark and Kafka in Java programming language. Experience with any NoSQL stores (HBase, Cassandra, MongoDB, Influx DB). Solid understanding of secure application development methodologies Experience in developing microservices using spring framework is a plus Capable of working as an individual contributor and within team too Design, build & maintain efficient, reusable & reliable code Experience in Hadoop based technologies Java, Hive, Pig, Map Reduce, Spark, python/Scala , Azure Should be able to understand complex architectures and be comfortable working with multiple teams Excellent communication, client engagement and client management skills are strongly preferred. Minimum Bachelors degree in Computer Science, Engineering, Business Information Systems, or related field. If the above profile suits you then request, to share your updated profile with below HR details: Full Name- Email Id- Phone No- Total years of experience - Relevant experience Bigdata- Relevant experience AWS- Relevant experience in Snowflake- Relevant experience in Redshift- Rating on SQL (out of5 )- Any other Technology- Notice period - CTC- ECTC- Current company- Current location: Preferred location: Any offers, If yes, Pls mention- Interview availability; Pls mention the Date and Time Revert with your confirmation

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models. Proficiency in below listed skills will be crucial in building and maintaining pipelines for training and inference datasets. Responsibilities: • Work in tandem with Data Scientists to design, develop, and implement machine learning pipelines. • Utilize PySpark for data processing, transformation, and preparation for model training. • Leverage AWS EMR and S3 for scalable and efficient data storage and processing. • Implement and manage ETL workflows using Streamsets for data ingestion and transformation. • Design and construct pipelines to deliver high-quality training and inference datasets. • Collaborate with cross-functional teams to ensure smooth deployment and real-time/near real-time inferencing capabilities. • Optimize and fine-tune pipelines for performance, scalability, and reliability. • Ensure IAM policies and permissions are appropriately configured for secure data access and management. • Implement Spark architecture and optimize Spark jobs for scalable data processing. Total Experience Expected: 04-06 years

As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution. Your primary responsibilities include: Lead the design and construction of new solutions using the latest technologies, always looking to add business value and meet user requirements. Strive for continuous improvements by testing the build solution and working under an agile framework. Discover and implement the latest technologies trends to maximize and build creative solutions Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Experience with Apache Spark (PySpark): In-depth knowledge of Sparks architecture, core APIs, and PySpark for distributed data processing. Big Data Technologies: Familiarity with Hadoop, HDFS, Kafka, and other big data tools. Data Engineering Skills: Strong understanding of ETL pipelines, data modeling, and data warehousing concepts. Strong proficiency in Python: Expertise in Python programming with a focus on data processing and manipulation. Data Processing Frameworks: Knowledge of data processing libraries such as Pandas, NumPy. SQL Proficiency: Experience writing optimized SQL queries for large-scale data analysis and transformation. Cloud Platforms: Experience working with cloud platforms like AWS, Azure, or GCP, including using cloud storage systems Preferred technical and professional experience Define, drive, and implement an architecture strategy and standards for end-to-end monitoring. Partner with the rest of the technology teams including application development, enterprise architecture, testing services, network engineering, Good to have detection and prevention tools for Company products and Platform and customer-facing

About Narayana Health: Narayana Health is headquartered in Bengaluru, India, and operates a network of hospitals in India and Overseas. Our mission is to deliver high quality, affordable healthcare services to the broader population. Narayana Health Group is Indias leading healthcare provider and one of the largest hospital groups in the country with a network of 21 hospitals, 5 heart centers, and 19 primary care facilities. The NH group treats over 2.6 Million patients every year from over 78 countries covering 30+ medical specialties. Our Centers of Excellence help in treating Adult & Pediatric patients and we have one of the largest transplant centers in India. We have a strong presence across 17 locations in India, and an overseas hospital in the Cayman Islands, USA. Two of our hospitals have international accreditation from the Joint Commission International (JCI) and 19 hospitals have domestic accreditation from National Accreditation Board for Hospitals (NABH). For more details, please refer to our website at: https://www.narayanahealth.org About Medha Analytics: Medha Analytics works to simplify healthcare by unleashing the potential of data. With a powerful tech-stack, interesting ideas and inspired co-workers, no dream is too big at Medha Analytics. For more details, please refer to our website at: https://www.medha.health Job Description: Understanding business processes, understanding the software systems for retrieval, prepping and modelling data to build/maintain modern data platform for supporting reports and Dashboards. Create and maintain optimal data pipeline architecture. Assemble large, complex data sets that meet functional/non-functional business requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure technologies. Work with stakeholders including the Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs. Work with data and analytics experts to strive for greater functionality in our data systems. Candidate Requirement: Education : B.E. or B.Tech Experience : 3 to 5 Yrs as Data Engineer Knowledge & Skills : Experience of working on any of the public cloud AWS/Azure/GCP Experience working on Python/Scala/Java, Big Data technologies like spark and Hadoop File System (HDFS). Strong hands-on experience in writing code in Pyspark/Python/Scala/Shell language using best practices. Experience on Performance tuning of Spark Jobs and cluster configurations Any one Knowledge of AWS services viz., EC2, EMR, S3, Lambda, EBS, IAM, Redshift, RDS, GLUE ETL is desired. Knowledge of Azure services viz., ADF, Databricks, Azure Synapse Analytics, ADLS Gen2,Azure Devops is desired. Proficient in writing highly optimized SQLs and algorithm for data processing Sound understanding of the cloud relational and non-relation database, their storage concepts, best practices and use case. Knowledge in RESTful Webservice, Microsoft SQL Server, MySQL Server, and Agile methodology is an advantage. DevOps toolchain experience is plus (Docker/Kubernetes/Ansible/Git) Strong analytical, problem-solving, and communication skills Excellent command of both written and spoken English. Should be able to Design, Develop, Deliver & maintain Data Infrastructures.

Min 5-8 years of experience in Hadoop/big data technologies. Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr). Hands-on experience with Python/Pyspark. Design, develop, and optimize ETL pipelines using Python and PySpark to process and transform large-scale datasets, ensuring performance and scalability on big data platforms. Implement big data solutions for Retail banking use cases such Risk analysis, Management Reporting (time series, Vintage curves, Executive summary) and regulatory reporting, while maintaining data accuracy and compliance standards. Collaborate with cross-functional teams to integrate data from various sources, troubleshoot production issues, and ensure efficient, reliable data processing operations.

Job Description Looking for candidates with strong experience in software development, especially in Big Data development technologies including Java/Python & Spark/Hive/Hadoop Qualifications: BE/B.Tech/MCA/MS-IT/CS/B.Sc/BCA or any other degrees in related fields Experience in working on Hadoop Distribution, good understanding of core concepts and best practices Good experience in building/tuning Spark pipelines in JAVA/Python Good experience in writing complex Hive queries to derive business critical insights Good Programming experience with Java/Python Understanding of Data Lake vs Data Warehousing concepts Experience with AWS Cloud, exposure to Lambda/EMR/Kinesis will be good to have Experience in NoSQL Technologies MongoDB, Dynamo DB Roles and Responsibilities: Design and implement solutions for problems arising out of large-scale data processing Attend/drive various architectural, design and status calls with multiple stakeholders Ensure end-to-end ownership of all tasks being aligned Design, build & maintain efficient, reusable & reliable code Test implementation, troubleshoot & correct problems Capable of working as an individual contributor and within team too Ensure high quality software development with complete documentation and traceability Fulfil organizational responsibilities (sharing knowledge & experience with other teams/ groups) Conduct technical training(s)/session(s), write whitepapers/case studies/blogs etc. Years Of Experience - 3 to 8 Years Locations - Gurgaon,Bangalore Mandatory Skills- Bigdata, Java/Python, Spark

Position Overview: As a Senior Software Engineer on the AI Engineering Team at Cotiviti, you will be a leading force in developing robust, scalable machine learning solutions for healthcare applications. This senior-level position involves significant responsibility, including leading design and development efforts, mentoring junior engineers, and ensuring the delivery of high-quality solutions. Basic Qualifications: Bachelors degree in Computer Science, Engineering, Math, or a related field, or equivalent experience 7+ years of experience with Hadoop tech stack (Spark, Kafka) Should have experience with batch processing on large scale data with Spark and real-time without Spark Proficiency in programming languages such as Scala or Python Extensive experience with Kafka and data streaming platforms Advanced knowledge of Data Bricks on AWS or similar cloud platforms Proven experience building and maintaining microservices Deep understanding of data architecture principles Experience leading design and development of large systems Proficiency with CI/CD tools like Jenkins Experience with Unix/Linux operating systems Familiarity with Agile processes and tools like Jira and Confluence Strong drive to learn and advocate for development best practices Strong knowledge on troubleshooting and optimizing Spark applications Preferred Qualifications: Experience with Data Bricks on Azure/AWS Experience with Kafka, DataStream/DataFrame/DataSet Advanced proficiency with containerization tools like Docker, Kubernetes Knowledge of machine learning frameworks and tools such as DataRobot, H2O, ML Flow Experience with big data tools like Spark, Scala, Oozie, Hive or similar Streaming technologies Kafka, SparkStreams, RabbitMQ Experience with Continuous Integration and Delivery, unit testing, and functional automation testing Having API development experience will be a good addition. Healthcare domain experience will be a plus. Responsibilities: Lead the development and implementation of machine learning solutions for healthcare applications Guide and mentor a team of developers and testers Collaborate with data scientists and other engineers to design and build scalable solutions Write, test, and maintain high-quality code along with Code coverage Lead design and code review sessions Troubleshoot and resolve complex technical issues Document your work and share knowledge with the team Advocate for and implement development best practices Train and mentor junior engineers and software engineers Who You Are: Curious: You are always looking to deepen your understanding of complex problems. Creative: You enjoy coming up with innovative solutions to difficult challenges. Practical: You focus on delivering solutions that have real-world applications and value. Focused: You maintain a clear vision of your goals and work diligently to achieve them. Determined: You are committed to contributing to the development of advanced machine learning capabilities.

5+ years of hands-on experience in designing, building and supporting Data Applications using Spark, Sqoop and Hive Bachelors or masters degree in Computer Science or related field Strong knowledge of working with large data sets and high-capacity big data processing platform Strong experience in Unix and Shell scripting Advanced knowledge of the Hadoop ecosystem and its components In-depth knowledge of Hive, Shell scripting, Python, Spark Ability to write MapReduce jobs Experience using Job Schedulers like Autosys Hands on experience in HiveQL Good knowledge on Hadoop Architecture and HDFS Strong knowledge of working with large data sets and high-capacity big data processing platform Strong experience in Unix and Shell scripting Experience with jenkins for Continuous Integration Experience using Source Code and Version Control Systems like Bitbucket, Git Good to have experience on Agile Development Responsibilities : Develop components, application interfaces, and solution enablers while ensuring principal architecture integrity is maintained Ensures solutions are well designed with maintainability/ease of integration and testing built-in from the outset Participates and guides team in estimating work necessary to realize a story/requirement through the software delivery lifecycle Responsible for developing and delivering complex software requirements to accomplish business goals Ensures that software is developed to meet functional, non-functional, and compliance requirements Codes solutions, Unit testing and ensure the solution can be integrated successfully into the overall application/system with clear robust, and well-tested interfaces Required Skills : Hadoop, Hive, HDFS, Spark, Python, Un

Manage ETL pipelines, data engineering operations, and cloud infrastructure Experience in configuring data exchange and transfer methods Experience in orchestrating ETL pipelines with multiple tasks, triggers, and dependencies Strong proficiency with Python and Apache Spark; intermediate or better proficiency with SQL; experience with AWS S3 and EC2, Databricks, Ability to communicate efficiently and translate ideas with technical stakeholders in IT and Data Science Passionate about designing data infrastructure and eager to contribute ideas to help build robust data platforms