141 Hdfs Jobs - Page 3

Number of Openings* 2 Approved ECMS RQ# * 527250 Duration of contract* 6 months Total Yrs. of Experience* 10+ Relevant Yrs. of experience* 8+ Detailed JD *(Roles and Responsibilities) Lead Consultant An experience Big data professional who will technically lead the team in design and development Responsible for team delivery Mandatory skills* HDFS, Ozone, Hive, Impala, Spark, Atlas, Ranger Kafka, Flink, Spark Streaming Python/PySpark Excellent communication Experienced in design of data landscape Desired skills* GraphQL, Venafi (Certificate Mgt), Collibra, Azure DevOps Domain* Telecom Approx. vendor billing rate* (INR/Month) Excluding service tax 11500 INR/day Work Location* Bangalore, India. Working from Office Background check process to be followed: * Before onboarding / After onboarding: * BGV Agency: * After onboarding Mode of Interview: Telephonic/Face to Face/Skype Interview* Skype/Teams. (There may be a F2F

Duration: 12Months Job Type: Contract Work Type: Onsite Job Description : Analyzes business requirements/processes and system integration points to determine appropriate technology solutions. Designs, codes, tests and documents applications based on system and user requirements. Requirements: 2-4 years of relevant IT experience in Data-Warehousing Technologies with excellent communication and Analytical skills Should possess the below skillset experience Informatica 9 or above as an ETL Tool Teradata/Oracle/SQL Server as Warehouse Database Very strong in SQL / Macros Should know Basic ~ Medium UNIX Commands Knowledge on Hadoop- HDFS, Hive, PIG and YARN Knowledge on ingestion tool - Stream sets Good to have knowledge on Spark and Kafka Exposure in scheduling tools like Control-M Excellent analytical and problem-solving skills is a must have Excellent communication skills (oral and written) Must be experienced in diverse industry and tools and data warehousing technologies. Responsibilities: Prepares flow charts and systems diagrams to assist in problem analysis. Responsible for preparing design documentation. Designs, codes, tests and debugs software according to the clients standards, policies and procedures. Codes, tests and documents programs according to system standards. Prepares test data for unit, string and parallel testing. Analyzes business needs and creates software solutions. Evaluates and recommends software and hardware solutions to meet user needs. Interacts with business users and I/T to define current and future application requirements. Executes schedules, costs and documentation to ensure project comes to successful conclusion. Initiates corrective action to stay on project schedules. May assist in orienting, training, assigning and checking the work of lower-level employees. Leads small to moderate budget projects. Knowledge and Skills: Possesses and applies a broad knowledge of application programming processes and procedures to the completion of complex assignments. Competent to analyze diverse and complex problems. Possesses and applies broad knowledge of principles of applications programming. Competent to work in most phases of applications programming. Beginning to lead small projects or starting to offer programming solutions at an advanced level. Knowledge includes advanced work on standard applications programs including coding, testing and debugging. Advanced ability to effectively troubleshoot program errors. Advanced understanding of how technology decisions relate to business needs. Mandatory Skills: Informatica 9 or above as an ETL Tool Teradata/Oracle/SQL Server as Warehouse Database Very strong in SQL / Macros Should have good knowledge on UNIX Commands Experience: Total Exp 2-3 Years Rel Exp 2 years

Position: Senior Software Engineer Location: Hyderabad Duration: 12Months Job Type: Contract Work Type: Onsite Job Description : Analyzes business requirements/processes and system integration points to determine appropriate technology solutions. Designs, codes, tests and documents applications based on system and user requirements. Requirements: 6-8 years of relevant IT experience in Data-Warehousing Technologies with excellent communication and Analytical skills Should possess the below skillset experience Informatica 9 or above as an ETL Tool Teradata/Oracle/SQL Server as Warehouse Database Very strong in SQL / Macros Should know Basic ~ Medium UNIX Commands Knowledge on Hadoop- HDFS, Hive, PIG and YARN Knowledge on ingestion tool - Stream sets Good to have knowledge on Spark and Kafka Exposure in scheduling tools like Control-M Excellent analytical and problem-solving skills is a must have Excellent communication skills (oral and written) Must be experienced in diverse industry and tools and data warehousing technologies. Responsibilities: Prepares flow charts and systems diagrams to assist in problem analysis. Responsible for preparing design documentation. Designs, codes, tests and debugs software according to the clients standards, policies and procedures. Codes, tests and documents programs according to system standards. Prepares test data for unit, string and parallel testing. Analyzes business needs and creates software solutions. Evaluates and recommends software and hardware solutions to meet user needs. Interacts with business users and I/T to define current and future application requirements. Executes schedules, costs and documentation to ensure project comes to successful conclusion. Initiates corrective action to stay on project schedules. May assist in orienting, training, assigning and checking the work of lower-level employees. Leads small to moderate budget projects. Knowledge and Skills : Possesses and applies a broad knowledge of application programming processes and procedures to the completion of complex assignments. Competent to analyze diverse and complex problems. Possesses and applies broad knowledge of principles of applications programming. Competent to work in most phases of applications programming. Beginning to lead small projects or starting to offer programming solutions at an advanced level. Knowledge includes advanced work on standard applications programs including coding, testing and debugging. Advanced ability to effectively troubleshoot program errors. Advanced understanding of how technology decisions relate to business needs. Mandatory Skills: Informatica 9 or above as an ETL Tool Teradata/Oracle/SQL Server as Warehouse Database Very strong in SQL / Macros Should have good knowledge on UNIX Command

Experience: 2+ yrs of experience in IT, with At least 1+ years of experience with cloud and system administration. At least 2 years of experience with and strong understanding of 'big data' technologies in Hadoop ecosystem Hive, HDFS, Map/Reduce, Flume, Pig, Cloudera, HBase Sqoop, Spark etc. Job Overview: Smartavya Analytica Private Limited is seeking an experienced Hadoop Administrator to manage and support our Hadoop ecosystem. The ideal candidate will have strong expertise in Hadoop cluster administration, excellent troubleshooting skills, and a proven track record of maintaining and optimizing Hadoop environments. Key Responsibilities: • Install, configure, and manage Hadoop clusters, including HDFS, YARN, Hive, HBase, and other ecosystem components. Monitor and manage Hadoop cluster performance, capacity, and security. Perform routine maintenance tasks such as upgrades, patching, and backups. Implement and maintain data ingestion processes using tools like Sqoop, Flume, and Kafka. Ensure high availability and disaster recovery of Hadoop clusters. Collaborate with development teams to understand requirements and provide appropriate Hadoop solutions. Troubleshoot and resolve issues related to the Hadoop ecosystem. Maintain documentation of Hadoop environment configurations, processes, and procedures. Requirement: • Experience in Installing, configuring and tuning Hadoop distributions. Hands on experience in Cloudera. Understanding of Hadoop design principals and factors that affect distributed system performance, including hardware and network considerations. Provide Infrastructure Recommendations, Capacity Planning, work load management. Develop utilities to monitor cluster better Ganglia, Nagios etc. Manage large clusters with huge volumes of data Perform Cluster maintenance tasks Create and removal of nodes, cluster monitoring and troubleshooting Manage and review Hadoop log files Install and implement security for Hadoop clusters Install Hadoop Updates, patches and version upgrades. Automate the same through scripts Point of Contact for Vendor escalation. Work with Hortonworks in resolving issues Should have Conceptual/working knowledge of basic data management concepts like ETL, Ref/Master data, Data quality, RDBMS Working knowledge of any scripting language like Shell, Python, Perl Should have experience in Orchestration & Deployment tools. Academic Qualification:

What the Candidate Will Need / Bonus Points ---- What the Candidate Will Do ---- Partner with engineers, analysts, and product managers to define technical solutions that support business goals Contribute to the architecture and implementation of distributed data systems and platforms Identify inefficiencies in data processing and proactively drive improvements in performance, reliability, and cost Serve as a thought leader and mentor in data engineering best practices across the organization ---- Basic Qualifications ---- 7+ years of hands-on experience in software engineering with a focus on data engineering Proficiency in at least one programming language such as Python, Java, or Scala Strong SQL skills and experience with large-scale data processing frameworks (e.g., Apache Spark, Flink, MapReduce, Presto) Demonstrated experience designing, implementing, and operating scalable ETL pipelines and data platforms Proven ability to work collaboratively across teams and communicate technical concepts to diverse stakeholders ---- Preferred Qualifications ---- Deep understanding of data warehousing concepts and data modeling best practices Hands-on experience with Hadoop ecosystem tools (e.g., Hive, HDFS, Oozie, Airflow, Spark, Presto) Familiarity with streaming technologies such as Kafka or Samza Expertise in performance optimization, query tuning, and resource-efficient data processing Strong problem-solving skills and a track record of owning systems from design to production We welcome people from all backgrounds who seek the opportunity to help build a future where everyone and everything can move independently. If you have the curiosity, passion, and collaborative spirit, work with us, and let's move the world forward, together. Offices continue to be central to collaboration and Uber's cultural identity. Unless formally approved to work fully remotely, Uber expects employees to spend at least half of their work time in their assigned office. For certain roles, such as those based at green-light hubs, employees are expected to be in-office for 100% of their time. Please speak with your recruiter to better understand in-office expectations for this role. .Accommodations may be available based on religious and/or medical conditions, or as required by applicable law. To request an accommodation, please reach out to .

Job Location: Chennai Notice Period: Immediate - 30 Days MAX Job Description: 5-12 Years of in Big Data & Data related technology experience Expert level understanding of distributed computing principles Expert level knowledge and experience in Apache Spark Hands on programming with Python Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming Experience with messaging systems, such as Kafka or RabbitMQ Good understanding of Big Data querying tools, such as Hive, and Impala Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files Good understanding of SQL queries, joins, stored procedures, relational schemas Experience with NoSQL databases, such as HBase, Cassandra, MongoDB Knowledge of ETL techniques and frameworks Performance tuning of Spark Jobs Experience with native Cloud data services AWS or AZURE Databricks Ability to lead a team efficiently Experience with designing and implementing Big data solutions Practitioner of AGILE methodology

Job Title: Big Data Administrator Location : Hyderabad (Weekly once WFO) Experience : 5+ Years Department : Data Engineering / IT Infrastructure Job Summary: We are seeking a Big Data Administrator with strong expertise in Linux systems, AWS infrastructure, and Big Data technologies. This role is ideal for someone experienced in managing large-scale Hadoop ecosystems in production, with a deep understanding of observability, performance tuning, and automation using tools like Terraform or Ansible. Key Responsibilities: Manage and maintain large-scale Big Data clusters (Cloudera, Hortonworks, or AWS EMR) Develop and support infrastructure as code using Terraform or Ansible Administer Hadoop ecosystem components HDFS, YARN, Hive (Tez, LLAP), Presto, Spark Implement and monitor observability tools like Prometheus, InfluxDB, Dynatrace, Grafana, Splunk Optimize SQL performance on Hive/Spark and understand query plans Automate cluster operations using Python (PySpark) or Shell scripting Support Data Analysts & Scientists with tools like JupyterHub, R-Studio, H2O, SAS Handle data in various formats ORC, Parquet, Avro Integrate with and support Kubernetes-based environments (if applicable) Collaborate across teams for deployments, monitoring, and troubleshooting Must-Have Skills: 5+ years in Linux system administration and AWS cloud infrastructure Experience with Cloudera, Hortonworks, or EMR in production Strong in Terraform / Ansible for automation Solid hands-on with HDFS, YARN, Hive, Spark, Presto Proficient in Python and Shell scripting Familiar with observability tools : Grafana, Prometheus, InfluxDB, Splunk, Dynatrace Familiarity with Active Directory , Windows VDI platforms (Citrix, AWS Workspaces) Nice-to-Have Skills: Experience with Airflow , Oozie Familiar with Pandas, Numpy, Scipy, PyTorch Prior use of Jenkins, Chef, Packer Comfortable reading code in Java, Scala, Python, R Qualifications: Bachelors or Masters degree in Computer Science, Information Technology, or a related field Strong communication, collaboration, and troubleshooting skills Ability to thrive in remote or hybrid work environments Please send your updated resume at komalikab@upwardiq.com

About Company Job Description Key responsibilities: 1. Understand, implement, and automate ETL pipelines with better industry standards 2. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, design infrastructure for greater scalability, etc 3. Developing, integrating, testing, and maintaining existing and new applications 4. Design, and create data pipelines (data lake / data warehouses) for real world energy analytical solutions 5. Expert-level proficiency in Python (preferred) for automating everyday tasks 6. Strong understanding and experience in distributed computing frameworks, particularly Spark, Spark-SQL, Kafka, Spark Streaming, Hive, Azure Databricks etc 7. Limited experience in using other leading cloud platforms preferably Azure. 8. Hands on experience on Azure data factory, logic app, Analysis service, Azure blob storage etc. 9. Ability to work in a team in an agile setting, familiarity with JIRA and clear understanding of how Git works 10. Must have 5-7 years of experience

Role : Data Engineer - Azure Synapse Analytics - Experience in Data engineering projects using Microsoft Azure platform (Min 2-3 projects) - Strong expertise in data engineering tools and storage such as Azure ADLS Gen2, Blob storage - Experience implementing automated Synapse pipelines - Ability to implement Synapse pipelines for data integration ETL/ELT using Synapse studio - Experience integrating Synapse notebooks and Data Flow - Should be able to troubleshoot pipelines - Strong T-SQL programming skills or with any other flavor of SQL - Experience working with high volume data, large objects - Experience working in DevOps environments integrated with GIT for version control and CI/CD pipeline. - Good understanding of data modelling for data warehouse and data marts - Should have experience on Big data components like HIVE, Sqoop, HDFS, Spark - Strong verbal and written communication skills - Ability to learn, contribute and grow in a fast phased environment.

Responsibilities for this position include : - Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies - Write and tune complex Java, MapReduce, Pig and Hive jobs - Adapt quickly to change in requirements and be willing to work with different technologies if required - Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important - Lead the effort to build, implement and support the data infrastructure - Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices - Manage team of data engineers (both full-time associates and/or third party resources) - Own the majority of deliverables for the Big Data team from a delivery perspective - Analyzes and confirms the integrity of source data to be evaluated - Leads in deployment and auditing models and attributes for accuracy Education for Lead Data Engineer : Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management Skills for Lead Data Engineer : Desired skills for lead data engineer include : - Python - Spark - Java - Hive - SQL - Hadoop architecture - Large scale search applications and building high volume data pipelines - Message queuing - NoSQL - Scala Desired experience for lead data engineer includes : - Experience in development utilizing C# .NET 4.5 - Experience in managing a live service that customers depend - Experience in coaching and managing other engineers - Be a team player and enjoy collaboration with other engineers and teams - Experience with software version management systems - Experience with task/bug tracking software.

Responsibilities for this position include : - Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies - Write and tune complex Java, MapReduce, Pig and Hive jobs - Adapt quickly to change in requirements and be willing to work with different technologies if required - Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important - Lead the effort to build, implement and support the data infrastructure - Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices - Manage team of data engineers (both full-time associates and/or third party resources) - Own the majority of deliverables for the Big Data team from a delivery perspective - Analyzes and confirms the integrity of source data to be evaluated - Leads in deployment and auditing models and attributes for accuracy Education for Lead Data Engineer : Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management Skills for Lead Data Engineer : Desired skills for lead data engineer include : - Python - Spark - Java - Hive - SQL - Hadoop architecture - Large scale search applications and building high volume data pipelines - Message queuing - NoSQL - Scala Desired experience for lead data engineer includes : - Experience in development utilizing C# .NET 4.5 - Experience in managing a live service that customers depend - Experience in coaching and managing other engineers - Be a team player and enjoy collaboration with other engineers and teams - Experience with software version management systems - Experience with task/bug tracking software Exp : 5-7yrs

Responsibilities for this position include : - Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies - Write and tune complex Java, MapReduce, Pig and Hive jobs - Adapt quickly to change in requirements and be willing to work with different technologies if required - Experience leading a Backend/Distributed Data Systems team while remaining hands-on is very important - Lead the effort to build, implement and support the data infrastructure - Manage the business intelligence team and vendor partners, ensuring to prioritize projects according to customer and internal needs, and develops top-quality dashboards using industry best practices - Manage team of data engineers (both full-time associates and/or third party resources) - Own the majority of deliverables for the Big Data team from a delivery perspective - Analyzes and confirms the integrity of source data to be evaluated - Leads in deployment and auditing models and attributes for accuracy Education for Lead Data Engineer : Have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Statistics, Education, Technical, Information Technology, Information Systems, Mathematics, Computer Engineering, Management Skills for Lead Data Engineer : Desired skills for lead data engineer include : - Python - Spark - Java - Hive - SQL - Hadoop architecture - Large scale search applications and building high volume data pipelines - Message queuing - NoSQL - Scala Desired experience for lead data engineer includes : - Experience in development utilizing C# .NET 4.5 - Experience in managing a live service that customers depend - Experience in coaching and managing other engineers - Be a team player and enjoy collaboration with other engineers and teams - Experience with software version management systems - Experience with task/bug tracking software Exp : 5-7yrs

Skills: 5+ years of experience with Java + Bigdata as minimum required skill . Java, Micorservices ,Sprintboot, API ,Bigdata-Hive, Spark,Pyspark Roles and Responsibilities Skills: 5+ years of experience with Java + Bigdata as minimum required skill . Java, Micorservices ,Sprintboot, API ,Bigdata-Hive, Spark,Pyspark

Key Responsibilities: Big Data Architecture: Design, develop, and maintain scalable and distributed data architectures capable of processing large volumes of data. Data Storage Solutions: Implement and optimize data storage solutions using technologies such as Hadoop , Spark , and PySpark . PySpark Development: Develop and implement efficient ETL processes using PySpark to extract, transform, and load large datasets. Performance Optimization: Optimize PySpark applications for better performance, scalability, and resource management. Qualifications: Proven experience as a Big Data Engineer with a strong focus on PySpark . Deep understanding of Big Data processing frameworks and technologies. Strong proficiency in PySpark for developing and optimizing ETL processes and data transformations. Experience with distributed computing and parallel processing. Ability to collaborate in a fast-paced, innovative environment. Required Skills Pyspark,Big Data, Python,

As AWS Data Engineer at organization, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives You will design and maintain scalable data pipelines using AWS data analytical resources, enabling efficient data processing and analytics. Key Responsibilities: - Highly experiences in developing ETL pipelines using AWS Glue and EMR with PySpark/Scala. - Utilize AWS services (S3, Glue, Lambda, EMR, Step Functions) for data solutions. - Design scalable data models for analytics and reporting. - Implement data validation, quality, and governance practices. - Optimize Spark jobs for cost and performance efficiency. - Automate ETL workflows with AWS Step Functions and Lambda. - Collaborate with data scientists and analysts on data needs. - Maintain documentation for data architecture and pipelines. - Experience with Open source bigdata file formats such as Iceberg or delta or Hundi - Desirable to have experience in terraforming AWS data analytical resources. Must-Have Skills: - AWS (S3, Glue, EMR Lambda, EMR), PySpark or Scala, SQL, ETL development. Good-to-Have Skills: - Snowflake, Cloudera Hadoop (HDFS, Hive, Impala), Iceberg

As AWS Data Engineer at organization, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives You will design and maintain scalable data pipelines using AWS data analytical resources, enabling efficient data processing and analytics. Key Responsibilities: - Highly experiences in developing ETL pipelines using AWS Glue and EMR with PySpark/Scala. - Utilize AWS services (S3, Glue, Lambda, EMR, Step Functions) for data solutions. - Design scalable data models for analytics and reporting. - Implement data validation, quality, and governance practices. - Optimize Spark jobs for cost and performance efficiency. - Automate ETL workflows with AWS Step Functions and Lambda. - Collaborate with data scientists and analysts on data needs. - Maintain documentation for data architecture and pipelines. - Experience with Open source bigdata file formats such as Iceberg or delta or Hundi - Desirable to have experience in terraforming AWS data analytical resources. Must-Have Skills: - AWS (S3, Glue, EMR Lambda, EMR), PySpark or Scala, SQL, ETL development. Good-to-Have Skills: - Snowflake, Cloudera Hadoop (HDFS, Hive, Impala), Iceberg

As AWS Data Engineer at organization, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives You will design and maintain scalable data pipelines using AWS data analytical resources, enabling efficient data processing and analytics. Key Responsibilities: - Highly experiences in developing ETL pipelines using AWS Glue and EMR with PySpark/Scala. - Utilize AWS services (S3, Glue, Lambda, EMR, Step Functions) for data solutions. - Design scalable data models for analytics and reporting. - Implement data validation, quality, and governance practices. - Optimize Spark jobs for cost and performance efficiency. - Automate ETL workflows with AWS Step Functions and Lambda. - Collaborate with data scientists and analysts on data needs. - Maintain documentation for data architecture and pipelines. - Experience with Open source bigdata file formats such as Iceberg or delta or Hundi - Desirable to have experience in terraforming AWS data analytical resources. Must-Have Skills: - AWS (S3, Glue, EMR Lambda, EMR), PySpark or Scala, SQL, ETL development. Good-to-Have Skills: - Snowflake, Cloudera Hadoop (HDFS, Hive, Impala), Iceberg

As AWS Data Engineer at organization, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives You will design and maintain scalable data pipelines using AWS data analytical resources, enabling efficient data processing and analytics. Key Responsibilities: - Highly experiences in developing ETL pipelines using AWS Glue and EMR with PySpark/Scala. - Utilize AWS services (S3, Glue, Lambda, EMR, Step Functions) for data solutions. - Design scalable data models for analytics and reporting. - Implement data validation, quality, and governance practices. - Optimize Spark jobs for cost and performance efficiency. - Automate ETL workflows with AWS Step Functions and Lambda. - Collaborate with data scientists and analysts on data needs. - Maintain documentation for data architecture and pipelines. - Experience with Open source bigdata file formats such as Iceberg or delta or Hundi - Desirable to have experience in terraforming AWS data analytical resources. Must-Have Skills: - AWS (S3, Glue, EMR Lambda, EMR), PySpark or Scala, SQL, ETL development. Good-to-Have Skills: - Snowflake, Cloudera Hadoop (HDFS, Hive, Impala), Iceberg

Job Type : Full-Time Location Gurgaon Experience – 5-10 years Role – Big Data Architect. About the Role: As a Big Data Engineer, you will play a critical role in integrating multiple data sources, designing scalable data workflows, and collaborating with data architects, scientists, and analysts to develop innovative solutions. You will work with rapidly evolving technologies to achieve strategic business goals. Must-Have Skills: 4+ year’s of mandatory experience with Big data. 4+ year’s mandatory experience in Apache Spark. Proficiency in Apache Spark, Hive on Tez, and Hadoop ecosystem components. Strong coding skills in Python & Pyspark. Experience building reusable components or frameworks using Spark Expertise in data ingestion from multiple sources using APIs, HDFS, and NiFi. Solid experience working with structured, unstructured, and semi-structured data formats (Text, JSON, Avro, Parquet, ORC, etc.). Experience with UNIX Bash scripting and databases like Postgres, MySQL and Oracle. Ability to design, develop, and evolve fault-tolerant distributed systems. Strong SQL skills, with expertise in Hive, Impala, Mongo and NoSQL databases. Hands-on with Git and CI/CD tools Experience with streaming data technologies (Kafka, Spark Streaming, Apache Flink, etc.). Proficient with HDFS, or similar data lake technologies Excellent problem-solving skills — you will be evaluated through coding rounds Key Responsibilities: Must be capable of handling existing or new Apache HDFS cluster having name node, data node & edge node commissioning & decommissioning. Work closely with data architects and analysts to design technical solutions. Integrate and ingest data from multiple source systems into big data environments. Develop end-to-end data transformations and workflows, ensuring logging and recovery mechanisms. Must able to troubleshoot spark job failures. Design and implement batch, real-time, and near-real-time data pipelines. Optimize Big Data transformations using Apache Spark, Hive, and Tez Work with Data Science teams to enhance actionable insights. Ensure seamless data integration and transformation across multiple systems.

Job: Software Developer (Java + Big Data) Location: Indore Years of experience: 5-7 years Requisition Description 1. Problem solving and analytical skills 2. Good verbal and written communication skills Roles and Responsibilities 1. Design and develop high performance, scale-able applications with Java + Bigdata as minimum required skill . Java, Microservices , Spring boot, API ,Bigdata-Hive, Spark, Pyspark 2. Build and maintain efficient data pipelines to process large volumes of structured and unstructured data. 3. Develop micro-services, API and distributed systems 4. Worked on Spark, HDFS, CEPh, Solr/Elastic search, Kafka, Deltalake 5. Mentor and Guide junior members

Senior Big Data Engineer Experience: 7-9 Years of Experience. Preferred location: Hyderabad Must have Skills: Bigdata, AWS cloud, Java/Scala/Python, Ci/CD Good to Have Skills: Relational Databases (any), No SQL databases (any), Microservices or Domain services or API gateways or similar, Containers (Docker, K8s, etc) Required Skills Big Data,Aws Cloud,CI/CD,Java/Scala/Python

Role Proficiency: Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others' developmental activities Outcomes: Interpret the application/feature/component design to develop the same in accordance with specifications. Code debug test document and communicate product/component/feature development stages. Validate results with user representatives; integrates and commissions the overall solution Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or creating own solutions Optimises efficiency cost and quality. Influence and improve customer satisfaction Set FAST goals for self/team; provide feedback to FAST goals of team members Measures of Outcomes: Adherence to engineering process and standards (coding standards) Adherence to project schedule / timelines Number of technical issues uncovered during the execution of the project Number of defects in the code Number of defects post delivery Number of non compliance issues On time completion of mandatory compliance trainings Outputs Expected: Code: Code as per design Follow coding standards templates and checklists Review code - for team and peers Documentation: Create/review templates checklists guidelines standards for design/process/development Create/review deliverable documents. Design documentation r and requirements test cases/results Configure: Define and govern configuration management plan Ensure compliance from the team Test: Review and create unit test cases scenarios and execution Review test plan created by testing team Provide clarifications to the testing team Domain relevance: Advise Software Developers on design and development of features and components with a deep understanding of the business problem being addressed for the client. Learn more about the customer domain identifying opportunities to provide valuable addition to customers Complete relevant domain certifications Manage Project: Manage delivery of modules and/or manage user stories Manage Defects: Perform defect RCA and mitigation Identify defect trends and take proactive measures to improve quality Estimate: Create and provide input for effort estimation for projects Manage knowledge: Consume and contribute to project related documents share point libraries and client universities Review the reusable documents created by the team Release: Execute and monitor release process Design: Contribute to creation of design (HLD LLD SAD)/architecture for Applications/Features/Business Components/Data Models Interface with Customer: Clarify requirements and provide guidance to development team Present design options to customers Conduct product demos Manage Team: Set FAST goals and provide feedback Understand aspirations of team members and provide guidance opportunities etc Ensure team is engaged in project Certifications: Take relevant domain/technology certification Skill Examples: Explain and communicate the design / development to the customer Perform and evaluate test results against product specifications Break down complex problems into logical components Develop user interfaces business software components Use data models Estimate time and effort required for developing / debugging features / components Perform and evaluate test in the customer or target environment Make quick decisions on technical/project related challenges Manage a Team mentor and handle people related issues in team Maintain high motivation levels and positive dynamics in the team. Interface with other teams designers and other parallel practices Set goals for self and team. Provide feedback to team members Create and articulate impactful technical presentations Follow high level of business etiquette in emails and other business communication Drive conference calls with customers addressing customer questions Proactively ask for and offer help Ability to work under pressure determine dependencies risks facilitate planning; handling multiple tasks. Build confidence with customers by meeting the deliverables on time with quality. Estimate time and effort resources required for developing / debugging features / components Make on appropriate utilization of Software / Hardware's. Strong analytical and problem-solving abilities Knowledge Examples: Appropriate software programs / modules Functional and technical designing Programming languages - proficient in multiple skill clusters DBMS Operating Systems and software platforms Software Development Life Cycle Agile - Scrum or Kanban Methods Integrated development environment (IDE) Rapid application development (RAD) Modelling technology and languages Interface definition languages (IDL) Knowledge of customer domain and deep understanding of sub domain where problem is solved Additional Comments: Essential - Analysis and problem-solving capabilities. - Degree in Maths/Physics/Statistics/similar discipline or previous experience of a similar role. - Ability to interpret data, recognize problems as they arise, suggest and implement appropriate actions. - Ability to communicate clearly, both verbally and via detailed specifications and reports. - Excellent organizational skills and capability of working to tight deadlines. - Ability to work to a high level of accuracy. - Experience in designing, developing, and maintaining applications using Python and Pyspark in a Hadoop environment with HDFS. - Experience in AWS services S3, Lambdas, Managed Airflow (MWAA) and EMR Serverless. Desirable - Experience using data dashboarding/presentation tools like PowerBI. - Knowledge of data processing tools like Snowflake. - Experience in using databases like MongoDB and DynamoDB - Any experience in working with object-oriented languages like Java or .NET would be a plus. - Experian in AWS services like Kinesis, API Gateway etc - Awareness of CI/CD tools like Jenkins or Harness Required Skills Python,Aws Services,Pyspark,Hadoop Platform

Job Title: Big Data Developer Project Support & Mentorship Location: Mumbai Employment Type: Full-Time/Contract Department: Engineering & Delivery Position Overview: We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent. You will work on live client projects, provide technical support, contribute to solution delivery, and serve as a go-to technical mentor for less experienced team members. Key Responsibilities: Perform hands-on Big Data development work, including coding, testing, troubleshooting, and deploying solutions. Support ongoing client projects, addressing technical challenges and ensuring smooth delivery. Collaborate with junior engineers to guide them on coding standards, best practices, debugging, and project execution. Review code and provide feedback to junior engineers to maintain high quality and scalable solutions. Assist in designing and implementing solutions using Hadoop, Spark, Hive, HDFS, and Kafka. Lead by example in object-oriented development, particularly using Scala and Java. Translate complex requirements into clear, actionable technical tasks for the team. Contribute to the development of ETL processes for integrating data from various sources. Document technical approaches, best practices, and workflows for knowledge sharing within the team. Required Skills and Qualifications: 8+ years of professional experience in Big Data development and engineering. Strong hands-on expertise with Hadoop, Hive, HDFS, Apache Spark, and Kafka. Solid object-oriented development experience with Scala and Java. Strong SQL skills with experience working with large data sets. Practical experience designing, installing, configuring, and supporting Big Data clusters. Deep understanding of ETL processes and data integration strategies. Proven experience mentoring or supporting junior engineers in a team setting. Strong problem-solving, troubleshooting, and analytical skills. Excellent communication and interpersonal skills. Preferred Qualifications: Professional certifications in Big Data technologies (Cloudera, Databricks, AWS Big Data Specialty, etc.). Experience with cloud Big Data platforms (AWS EMR, Azure HDInsight, or GCP Dataproc). Exposure to Agile or DevOps practices in Big Data project environments. What We Offer: Opportunity to work on challenging, high-impact Big Data projects. Leadership role in shaping and mentoring the next generation of engineers. Supportive and collaborative team culture. Flexible working environment Competitive compensation and professional growth opportunities.

Position : Data Engineer Experience: 4+ years Availability: Immediate joiners preferred Work Mode: Work from Office (WFO) Location: MIDC, Ghansoli, Navi Mumbai Working Days: 5 days a week (Client Location) About the Role: We are seeking a skilled and detail-oriented Data Engineer to join our growing team. The ideal candidate will have strong proficiency in Python, SQL and Databricks and experience in building and maintaining scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other engineers to ensure data availability, reliability, and quality across our systems. Required Skills & Qualifications: Experience with Databricks - Mandatory Proficiency with open-Source technologies like HDFS, Hive, Kafka, Spark - Mandatory Ability to solve any ongoing issues with operating the cluster - Mandatory Expert in SQL Experience in Python Experience with building stream-processing systems Optional

6+ years of total IT experience 3+ years of experience with Hadoop (Cloudera)/big data technologies Knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr) Experience in designing and developing Data Pipelines for Data Ingestion or Transformation using Java Scala or Python. Experience with Spark programming (Pyspark, Scala, or Java) Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required. Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus. Hand on experience in CI/CD, Scheduling and Scripting Ensure automation through CI/CD across platforms both in cloud and on-premises System level understanding - Data structures, algorithms, distributed storage & compute Can-do attitude on solving complex business problems, good interpersonal and teamwork skills