344 Hdfs Jobs - Page 3

Expertise in Hadoop ecosystem, PySpark, Python,SQL,develop,maintain scalable data pipelines ETL workflows.Ensuring data quality,optimizing performance,collaborating stakeholders,managing HDFS/Hive/HBase,supporting data-driven cloud (GCP preferred).

• Big Data/Hadoop Experience particularly in ingesting data and implementing Data ingestion pipelines, SQOOP, HADOOP, HDFS, HIVE, IMPALA, Java, Scala, Spark • Scala is Mandatory

As a skilled professional with over 7 years of experience, you will be responsible for reviewing and understanding business requirements to ensure timely completion of development tasks with rigorous testing to minimize defects. Collaborating with a software development team is crucial to implement best practices and enhance the performance of Data applications, meeting client needs effectively. In this role, you will collaborate with various teams within the company and engage with customers to comprehend, translate, define, and design innovative solutions for their business challenges. Your tasks will also involve researching new Big Data technologies to evaluate their maturity and alignment with business and technology strategies. Operating within a rapid and agile development process, you will focus on accelerating speed to market while upholding necessary controls. Your qualifications should include a BE/B.Tech/MCA degree with a minimum of 6 years of IT experience, including 4 years of hands-on experience in design and development using the Hadoop technology stack and various programming languages. Furthermore, you are expected to have proficiency in multiple areas such as Hadoop, HDFS, MR, Spark Streaming, Spark SQL, Spark ML, Kafka/Flume, Apache NiFi, Hortonworks Data Platform, Hive, Pig, Sqoop, NoSQL Databases (HBase, Cassandra, Neo4j, MongoDB), Visualization & Reporting frameworks (D3.js, Zeppelin, Grafana, Kibana, Tableau, Pentaho), Scrapy for web crawling, Elastic Search, Google Analytics data streaming, and Data security protocols (Kerberos, Open LDAP, Knox, Ranger). A strong knowledge of the current technology landscape, industry trends, and experience in Big Data integration with Metadata Management, Data Quality, Master Data Management solutions, structured/unstructured data is essential. Your active participation in the community through articles, blogs, or speaking engagements at conferences will be highly valued in this role.,

As a Software Engineer with expertise in big data technologies, you will be responsible for working with technologies such as Hadoop, Hive, and Spark to process and analyze large datasets. You will collaborate with cross-functional teams to gather requirements and implement solutions that align with business needs. Additionally, you will participate in all phases of the software development lifecycle, including planning, development, testing, and deployment. Your role will also involve implementing CI/CD pipelines using tools like Jenkins to automate build, test, and deployment processes. You should be adaptable to new technologies and methodologies, demonstrating a continuous learning mindset. Troubleshooting and resolving issues in both production and development environments will be a key part of your responsibilities. Hands-on experience in cloud platforms such as Azure or AWS will also be beneficial for this role. We are looking for a talented and motivated individual with 2-5 years of experience in software development, specifically in big data frameworks like Spark, Scala, Hadoop, Hive, and HDFS. Familiarity with CI/CD and DevOps practices and tools, such as Maven, Jenkins, and GIT, is required. Experience working in an agile team setting is also essential. Basic knowledge in Python and JFrog artifactory would be considered as additional skills. Joining our dynamic team at Socit Gnrale will provide you with the opportunity to make a positive impact on the future. Our company values innovation, collaboration, and continuous development. Employees have the chance to engage in solidarity actions, such as sponsoring individuals struggling with their professional integration or sharing skills with charities. Diversity and inclusion are at the core of our values, and we welcome individuals who are looking to grow in a stimulating and caring environment.,

Job role - Hadoop developer Experience - 2 to 10 years Location - PAN INDIA This Job Opportunity is for TOP Leading MNCs / Permanent role . Role- Hadoop Developer / Module Lead Technical skills- Hadoop, Spark, Scala, Impala, Hive, Kafka Must-Have Extensive experience and hands on implementation experience with Spark, Scala, Impala, Hive, Kafka, SQOOP Good-to-Have Extensive knowledge with Data frames, Data Sets and RDDs.

As a Big Data/Hadoop Trainer, your main responsibility will be conducting training sessions on Big Data and Hadoop. You will be tasked with creating assignments and projects that are centered around Hadoop to enhance the learning experience of the participants. To excel in this role, you should have a minimum of 2 years of hands-on experience in Hadoop/Big Data Technology within the corporate sector. Your profile should showcase excellent knowledge of Hadoop, Big Data, HDFS, MapReduce, Pig, Hive, Sqoop, ZooKeeper, Hbase, and Java. Additionally, possessing strong communication and presentation skills is essential in effectively delivering the training content. We are looking for candidates with dynamic personalities who can engage and inspire the participants during the training sessions. If you meet these criteria and are passionate about sharing your knowledge in Big Data and Hadoop, we have weekend positions available for working faculties that could be a perfect fit for you.,

Were Hiring: Tech Lead – Big Data Technologies Location: Pune | Work Mode: Work from Office (WFO) Company: Leading MNC | Immediate Joiners Preferred Are you a Big Data expert with 6–9 years of experience and a passion for leading high-impact projects? We’re looking for a Tech Lead skilled in: Hadoop, Spark, Hive, Kafka, Sqoop, Oozie, Flume Java/Scala, SQL/NoSQL (PostgreSQL, MongoDB) System monitoring tools (Grafana, Ganglia), scripting & automation Role involves end-to-end ownership of data pipeline architecture, code reviews, performance tuning, and leading a talented team.

Unison Consulting Pte Ltd is looking for a skilled Big Data Developer to join the team. In this role, you will be responsible for creating and maintaining big data solutions to support data-driven decision-making and analytics for our clients. You will work with both structured and unstructured data using cutting-edge technologies and platforms. As a Big Data Developer, your responsibilities will include designing, developing, and maintaining big data solutions utilizing technologies like Hadoop, HDFS, Spark, and Hive. You will be tasked with handling large volumes of data from various sources, implementing efficient ETL processes, developing data models and governance frameworks, and conducting data exploration and validation. Collaboration with data scientists, analysts, and business stakeholders to deliver actionable insights will also be a key part of your role. Ideal candidates should possess a Bachelor's or Master's degree in Computer Science or a related field, along with proven experience as a Big Data Developer. Proficiency in technologies such as Hadoop, Spark, HDFS, and Hive is required, as well as strong programming skills in languages like Java, Scala, or Python. Experience with ETL tools, SQL, NoSQL databases, and data modeling is also necessary. Strong problem-solving skills, effective communication, and the ability to work well in cross-functional teams are essential for success in this role. If you thrive in a fast-paced environment, have a keen eye for detail, and are eager to continuously learn and adapt to new technologies, we would love to hear from you. Join us at Unison Consulting Pte Ltd and be part of a dynamic team dedicated to leveraging big data for impactful insights and solutions.,

As a Data Engineer, you will be responsible for designing and developing scalable data engineering solutions using Google Cloud Platform (GCP) and PySpark. Your main focus will be on optimizing Spark jobs for performance, scalability, and efficient resource utilization. You will also be involved in developing, maintaining, and enhancing ETL pipelines using BigQuery, Apache Airflow, and Cloud Composer. Collaborating with data scientists, analysts, and DevOps teams to translate business requirements into technical solutions will be a key aspect of your role. Ensuring data integrity and security by implementing data governance, compliance, and security best practices will be crucial. Monitoring production workloads, troubleshooting performance issues, and implementing enhancements will also be part of your responsibilities. You will be expected to implement and enforce coding standards, best practices, and performance tuning strategies. Additionally, supporting migration activities from on-premises data warehouses to GCP-based solutions will be part of your duties. Mentoring junior developers and contributing to knowledge-sharing within the team will be essential. Staying up to date with emerging cloud technologies, tools, and best practices in the data engineering ecosystem is also a key requirement. The ideal candidate should have + years of total experience in data engineering with + years of hands-on experience with Google Cloud Platform (GCP), including BigQuery, Apache Airflow, and Cloud Composer. Strong expertise in developing and optimizing large-scale data processing solutions using PySpark and Python is necessary. In-depth knowledge of SQL for data transformation and performance optimization is also required. Proficiency in big data technologies such as Hadoop, HDFS, Hive, and YARN is essential. Experience with distributed computing principles, data partitioning, and fault tolerance is preferred. Hands-on experience with CI/CD pipelines, version control (Git), and automation tools is a plus. Strong problem-solving, analytical, and troubleshooting skills are important for this role. Experience working in Agile/Scrum environments is beneficial. Excellent communication and collaboration skills are necessary to work with offshore and onshore teams. In this role, you will have the opportunity to work in a collaborative environment where your contributions create value. You will develop innovative solutions, build lasting relationships with colleagues and clients, and have access to global capabilities to bring your ideas to life. Your career will evolve in a company built to grow and last, supported by leaders who care about your well-being and provide opportunities for skill development and growth. Join us, one of the world's largest IT and management consulting companies, and become a part of a team dedicated to making a difference in the technology industry.,

Company Description Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. With a mission to fuel bold visions, Blend tackles significant challenges by seamlessly aligning human expertise with artificial intelligence. The company is dedicated to unlocking value and fostering innovation for its clients by harnessing world-class people and data-driven strategy. We believe that the power of people and AI can have a meaningful impact on your world, creating more fulfilling work and projects for our people and clients. For more information, visit www.blend360.com Job Description We are looking for a forward-thinking Data & AI Engineer with 13 years of experience in data engineering and a passion for using modern AI tools to accelerate development workflows. The ideal candidate is proficient in Python, SQL, PySpark, and has experience working in on-premise big data environments (e.g., Spark, Hadoop, Hive, HDFS). This role is ideal for someone eager to blend traditional data engineering practices with AI-augmented software development, helping us build high-performance pipelines and deliver faster, smarter solutions. What Youll Be Doing Develop and maintain robust ETL/ELT pipelines using Python, SQL, and PySpark. Work with on-premise big data platforms such as Spark, Hadoop, Hive, and HDFS. Optimize and troubleshoot workflows to ensure performance, reliability, and quality. Use AI tools to assist with code generation, testing, debugging, and documentation. Collaborate with data scientists, analysts, and engineers to support data-driven use cases. Maintain up-to-date documentation using AI summarization tools. Apply AI-augmented software engineering practices, including automated testing, code reviews, and CI/CD. Identify opportunities for automation and process improvement across the data lifecycle. Qualifications 13 years of hands-on experience as a Data Engineer or in a similar data-focused engineering role. Proficiency in Python for data manipulation, automation, and scripting. Solid understanding of SQL and relational database design. Experience building distributed data processing solutions with PySpark. Familiarity with on-premise big data ecosystems, including Hadoop, Hive, HDFS. Active use of AI development tools, such as: GitHub Copilot, Windsurf, Cursor for intelligent code assistance ChatGPT or similar for testing support, refactoring, and documentation AI-based testing frameworks or custom scripts Familiar with Git and CI/CD pipelines. Strong analytical skills and a mindset for automation and innovation. Show more Show less

Company Description Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. With a mission to fuel bold visions, Blend tackles significant challenges by seamlessly aligning human expertise with artificial intelligence. The company is dedicated to unlocking value and fostering innovation for its clients by harnessing world-class people and data-driven strategy. We believe that the power of people and AI can have a meaningful impact on your world, creating more fulfilling work and projects for our people and clients. For more information, visit www.blend360.com Job Description We are looking for a forward-thinking Data & AI Engineer with 13 years of experience in data engineering and a passion for using modern AI tools to accelerate development workflows. The ideal candidate is proficient in Python, SQL, PySpark, and has experience working in on-premise big data environments (e.g., Spark, Hadoop, Hive, HDFS). This role is ideal for someone eager to blend traditional data engineering practices with AI-augmented software development, helping us build high-performance pipelines and deliver faster, smarter solutions. What Youll Be Doing Develop and maintain robust ETL/ELT pipelines using Python, SQL, and PySpark. Work with on-premise big data platforms such as Spark, Hadoop, Hive, and HDFS. Optimize and troubleshoot workflows to ensure performance, reliability, and quality. Use AI tools to assist with code generation, testing, debugging, and documentation. Collaborate with data scientists, analysts, and engineers to support data-driven use cases. Maintain up-to-date documentation using AI summarization tools. Apply AI-augmented software engineering practices, including automated testing, code reviews, and CI/CD. Identify opportunities for automation and process improvement across the data lifecycle. Qualifications 13 years of hands-on experience as a Data Engineer or in a similar data-focused engineering role. Proficiency in Python for data manipulation, automation, and scripting. Solid understanding of SQL and relational database design. Experience building distributed data processing solutions with PySpark. Familiarity with on-premise big data ecosystems, including Hadoop, Hive, HDFS. Active use of AI development tools, such as: GitHub Copilot, Windsurf, Cursor for intelligent code assistance ChatGPT or similar for testing support, refactoring, and documentation AI-based testing frameworks or custom scripts Familiar with Git and CI/CD pipelines. Strong analytical skills and a mindset for automation and innovation. Show more Show less

Organization: At CommBank, we never lose sight of the role we play in other people's financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and energy all contribute to the impact that we can make with our work. Together we can achieve great things. Job Title: Sr. Data Engineer Location: Bangalore - Manyata Tech Park Business & Team: RBS (HB Domain) ---Search & Find Impact & contribution: You will play a pivotal role in driving business outcomes through effective data management and analysis. Enhancing Decision Making with Data Ensuring Data Security and Compliance Future Trends in Data Engineering Roles & Responsibilities: Design and develop highly reliable and scalable data pipelines and data platforms with comprehensive test coverage. Collaborate with stakeholders to analyse and translate requirements to technical implementation. Identify and drive opportunities for continuous improvement within the team and in delivery of products. Provide mentoring and technical assistance to other members of the team, including more junior Data Engineers. Produce high quality, sustainable solutions to meet business requirements, leveraging approved delivery frameworks and by applying industry best practice. Provide technical governance of product delivery to ensure successful delivery and adoption. Deliver data engineering solutions aligned to core concepts of data design, preparation, transformation, and load. Build and implement data pipelines in distributed data platforms including warehouses, databases, data lakes and cloud lake houses to enable data predictions and models, and reporting and visualisation analysis via data integration tools and frameworks. Contribute to domain planning, providing guidance to ensure that technical deliveries are aligned to engineering direction and strategy. Essential Skills: 8-12 years experience of relevant experience Hadoop, Spark, Scala/Python, Hive, SQL, AWS(Cloud), HDFS . Education Qualifications: Bachelor's degree in engineering in Computer Science/Information Technology If you're already part of the Commonwealth Bank Group (including Bankwest, x15ventures), you'll need to apply through to submit a valid application. We're keen to support you with the next step in your career. We're aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696. Advertising End Date: 14/09/2025

Organization: At CommBank, we never lose sight of the role we play in other people's financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and energy all contribute to the impact that we can make with our work. Together we can achieve great things. Job Title: Staff. Data Engineer Location: Bangalore - Manyata Tech Park Business & Team: RBS (HB Domain) ---Search & Find Impact & contribution: You will play a pivotal role in driving business outcomes through effective data management and analysis. Enhancing Decision Making with Data Ensuring Data Security and Compliance Future Trends in Data Engineering Roles & Responsibilities: Design and develop highly reliable and scalable data pipelines and data platforms with comprehensive test coverage. Collaborate with stakeholders to analyse and translate requirements to technical implementation. Identify and drive opportunities for continuous improvement within the team and in delivery of products. Provide mentoring and technical assistance to other members of the team, including more junior Data Engineers. Produce high quality, sustainable solutions to meet business requirements, leveraging approved delivery frameworks and by applying industry best practice. Provide technical governance of product delivery to ensure successful delivery and adoption. Deliver data engineering solutions aligned to core concepts of data design, preparation, transformation, and load. Build and implement data pipelines in distributed data platforms including warehouses, databases, data lakes and cloud lake houses to enable data predictions and models, and reporting and visualisation analysis via data integration tools and frameworks. Contribute to domain planning, providing guidance to ensure that technical deliveries are aligned to engineering direction and strategy. Essential Skills: 12-15 years experience of relevant experience Hadoop, Spark, Scala/Python, Hive, SQL, AWS(Cloud), HDFS . Education Qualifications: Bachelor's degree in engineering in Computer Science/Information Technology If you're already part of the Commonwealth Bank Group (including Bankwest, x15ventures), you'll need to apply through to submit a valid application. We're keen to support you with the next step in your career. We're aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696. Advertising End Date: 14/09/2025

Data Software Engineer Job Description: 1. 5-12 Years of in Big Data & Data related technology experience 2. Expert level understanding of distributed computing principles 3. Expert level knowledge and experience in Apache Spark 4. Hands on programming with Python 5. Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop 6. Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming 7. Experience with messaging systems, such as Kafka or RabbitMQ 8. Good understanding of Big Data querying tools, such as Hive, and Impala 9. Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files 10. Good understanding of SQL queries, joins, stored procedures, relational schemas 11. Experience with NoSQL databases, such as HBase, Cassandra, MongoDB 12. Knowledge of ETL techniques and frameworks 13. Performance tuning of Spark Jobs 14. Experience with native Cloud data services AWS or AZURE Databricks or GCP 15. Ability to lead a team efficiently 16. Experience with designing and implementing Big data solutions 17. Practitioner of AGILE methodology

Times Internet is the largest digital products company and the digital venture of Times of India, India's largest media and entertainment group. TIL websites are among the fastest growing Web / Mobile based networks worldwide. Since its inception in 1999, Times Internet has led the Internet revolution in India and has emerged as India's foremost web entity, running diverse portals and niche websites. The Database Professional will participate in the design, implementation, automation, optimization, and ongoing operational administration related tasks backend system running on MySQL/MariaDB/ MongoDB/RedisDB/Hadoop database infrastructure. Candidates should be ready to take the instant support/operational challenges of infrastructure platforms running database services. Job Specifications: - Hands-on experience on MySQL/MariaDB RDBMS and related tools like Percona Xtrabackup and Percona-tool-kit - Hands-on experience on MongoDB NoSQL and Cache(RedisDB) datastores - Hands-on experience working on Private as well Public IT Infrastructure clouds like AWS - Hands-on experience on the optimization of database SQL or No-SQL queries - Should have working experience on implementing best optimized and secure practices for RDS, NoSQL, and Cache database stores - Should have working experience on infra-resource planning, database upgrades/patching, backup/recovery, and database troubleshooting - Should have working experience on all database support activities like Replication, Scalability, Availability, Performance tuning/optimizations on the database servers running large volumes of data - Should have working experience on building highly scalable schema design for the application using No-SQL, RDBMS, and Cache databases - Good to have experience on Hadoop Technologies like HDFS, Hive, Flume, Sqoop, Spark etc.,

Job Description: About Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities, and shareholders every day. One of the keys to driving Responsible Growth is being a great place to work for our teammates around the world. We're devoted to being a diverse and inclusive workplace for everyone. We hire individuals with a broad range of backgrounds and experiences and invest heavily in our teammates and their families by offering competitive benefits to support their physical, emotional, and financial well-being. Bank of America believes both in the importance of working together and offering flexibility to our employees. We use a multi-faceted approach for flexibility, depending on the various roles in our organization. Working at Bank of America will give you a great career with opportunities to learn, grow and make an impact, along with the power to make a difference. Join us! Global Business Services Global Business Services delivers Technology and Operations capabilities to Lines of Business and Staff Support Functions of Bank of America through a centrally managed, globally integrated delivery model and globally resilient operations. Global Business Services is recognized for flawless execution, sound risk management, operational resiliency, operational excellence, and innovation. In India, we are present in five locations and operate as BA Continuum India Private Limited (BACI), a non-banking subsidiary of Bank of America Corporation and the operating company for India operations of Global Business Services. Process Overview. The Data Analytics Strategy platform and decision tool team is responsible for Data strategy for entire CSWT and development of platforms which supports the Data Strategy. Data Science platform, Graph Data Platform, Enterprise Events Hub are key platforms of Data Platform initiative. Job Description. As a Senior Hadoop Developer to develop Hadoop components in SDP (strategic data platform), individual will be responsible for understanding design, propose high level and detailed design solutions, and ensure that coding practices/quality comply with software development standards. Working as an individual contributor in projects, person should have good analytical skills to take a quick decision during the tough times, Person should have good knowledge writing complex queries in a larger cluster. Engage in discussions with architecture teams for coming out with design solutions, proposing new technology adoption ideas, attending project meetings, partnering with near shore and offshore teammates in an agile environment, coordinating with other application teams, development, testing, upstream and downstream partners, etc. Responsibilities: Develop high-performance and scalable Analytics solutions using the Big Data platform to facilitate the collection, storage, and analysis of massive data sets from multiple channels. Develop efficient utilities, data pipelines, ingestion frameworks that can be utilized across multiple business areas. Utilize your in-depth knowledge of Hadoop stack and storage technologies, including HDFS, Spark, MapReduce, Yarn, Hive, Sqoop, Impala, Hue, and Oozie, to design and optimize data processing workflows. Data analysis, coding, Performance Tunning, propose improvement ideas, drive the development activities at offshore. Analyze complex Hive Queries, able to modify Hive queries, tune Hive Queries Hands on experiences writing scripts in python/shell scripts and modify scripts. Provide guidance and mentorship to junior teammates. Work with the strategic partners to understand the requirements work on high level & detailed design to address the real time issues in production. Partnering with near shore and offshore teammates in Agile environment, coordinating with other application teams, development, testing, up/down stream partners, etc. Hands on experiences writing scripts in python/shell scripts and modify scripts. Work on multiple projects concurrently, take ownership & pride in the work done by them, attending project meetings, understanding requirements, designing solutions, developing code. Identify gaps in technology and propose viable solutions. Identify improvement areas within the application and work with the respective teams to implement the same. Ensuring adherence to defined process & quality standards, best practices, high quality levels in all deliverables. Desired Skills. Data Lake Architecture: Understanding of Medallion architecture ingestion Frameworks: Knowledge of ingestion frameworks like structured, unstructured, and semi structured Data Warehouse: Familiarity with Apache Hive and Impala Performs Continuous Integration and Continuous Development (CI-CD) activities. Hands on experience working in a Cloudera data platform (CDP) to support the Data Science Contributes to story refinement and definition of requirements. Participates in estimating work necessary to realize a story/requirement through the delivery lifecycle. Extensive hands-on supporting platforms to allow modelling and analysts go through the complete model lifecycle management (data munging, model develop/train, governance, deployment) Experience with model deployment, scoring and monitoring for batch and real-time on various technologies and platforms. Experience in Hadoop cluster and integration includes ETL, streaming and API styles of integration. Experience in automation for deployment using Ansible Playbooks, scripting. Experience with developing and building RESTful API services in an efficient and scalable manner. Design and build and deploy streaming and batch data pipelines capable of processing and storing large datasets quickly and reliably using Kafka, Spark and YARNfor large volumes of data (TBs) Experience with processing and deployment technologies such YARN, Kubernetes /Containers and Serverless Compute for model development and training. Effective communication, Strong stakeholder engagement skills, Proven ability in leading and mentoring a team of software engineers in a dynamic environment. Requirements. Education. Graduation / Post Graduation Experience Range. 7 to 12 years Foundational Skills Hadoop, Hive, Sqoop, Impala, Unix/Linux scripts. Desired Skills Python, CI/CD, ETL. Work Timings. 11:30 AM to 8:30 PM IST Job Location. Chennai / Hyderabad

As a Python Senior Developer, you should have a minimum of 5 years of experience in Python programming. You must possess extensive experience in handling complex projects and demonstrate strong leadership skills to lead projects independently. Your effective communication and teamwork skills will be crucial in collaborating with team members. Your role will require good analytical skills and proficiency in web-based development technologies. It is essential to have expertise in Agile development methodology and strong problem-solving abilities. You should also possess a deep knowledge of Design patterns, Security, Performance tuning, and App monitoring. Experience in API Design, security patterns, and Microservices architecture is necessary. You must have a proven track record of building scalable, low-latency web applications using Flask, FastAPI, or similar frameworks. Additionally, experience in building Python-based SDKs and ORM frameworks is required. Knowledge in distributed in-memory systems like Redis, MemCache, GemFire, and SQL and no-SQL databases is essential. Understanding how to build production-grade applications using WSGI/ASGI frameworks such as Gunicorn, Uvicorn is a must. Experience with Jenkins, CICD pipeline, Logging framework, Splunk, ELK stack is considered beneficial. It would be advantageous to have familiarity with virtualization and cloud platforms such as Kubernetes, AWS, OpenShift, Docker Containers. Proficiency in version control systems like Git and developing test suites, application debugging skills is highly preferred. Good to have skills include an understanding of front-end technologies like JavaScript, HTML5, Angular, and Event-driven programming using Python. Knowledge of Generative AI technologies, LLMs, Embeddings, and building RAG-based solutions would be an added advantage. Familiarity with HDFS, Hive, Impala, PySpark, and other components of BigData systems is also beneficial for this role.,

As the Head of Technology - New Initiatives at Matrimony.com, you will be at the forefront of driving innovation in our WedTech domain through a groundbreaking project. Your role will be pivotal in orchestrating the convergence of cutting-edge technology with matrimonial services, leading a minimum of one product tech team towards technological supremacy. Your key responsibilities will include: - Developing and implementing a strategic tech roadmap that aligns with industry advancements and business benchmarks. - Overseeing the product engineering lifecycle of our flagship product, ensuring scalability and efficiency from prototype to production readiness. - Building and nurturing a high-caliber tech team, fostering a culture of innovation and problem-solving. - Managing a fresh tech stack including Node.js, React, Kubernetes, AWS/Azure/GCP, GraphQL, and Docker, and integrating next-gen technologies. - Collaborating with cross-functional teams, external partners, and the global tech community to translate technical concepts into actionable plans. - Establishing top-tier practices for data protection, compliance, and system resilience while staying abreast of tech trends to mitigate potential challenges. Your skills should encompass: - Proficiency in coding languages such as NodeJS, PHP, Python, and Java. - Expertise in web and mobile architectures, frontend technologies like HTML, CSS, TypeScript, JavaScript, and frameworks like Ionic, Flutter, ReactJS, ReactNative, and VueJs. - Experience with REST and GraphQL for API development, data engineering, DevSecOps practices, cloud technologies, database management, code optimization, and testing methodologies. The ideal candidate will possess: - A track record of deploying disruptive digital solutions and scaling mobile apps and websites. - Demonstrated leadership skills in guiding tech teams towards tangible achievements. - A minimum of 8-10 years of experience in tech leadership within a dynamic ecosystem. - Enthusiasm for innovation and a knack for tackling tech challenges. - A bachelor's degree with over 15 years of experience in technical and people management roles. This position is based in Chennai, and you will be joining Matrimony.com Ltd, India's first pure-play Wed-Tech Company, known for its flagship brand Bharat Matrimony. With a commitment to innovation and growth, Matrimony.com aims to become a billion-dollar revenue company with a lasting legacy.,

You are an experienced PySpark ETL Lead responsible for driving data integration efforts in analytics and data warehousing projects. Your role includes developing and managing PySpark scripts, creating ETL workflows, and ensuring efficient data processing and integration across systems. You should be strong in PySpark and Python for ETL and data processing, with experience in ETL pipeline design, data integration, and data warehousing. Proficiency in SQL and working with large datasets is required, along with familiarity with workflow schedulers such as Airflow and Cron. Hands-on experience with Big Data tools like Hive, HDFS, and Spark SQL is essential. It would be beneficial if you have experience with cloud platforms like AWS, Azure, or GCP. Your responsibilities will involve leading ETL development using PySpark, designing and scheduling data workflows, optimizing data processing performance, and collaborating with cross-functional teams. If you have a passion for data integration, a knack for optimizing processes, and enjoy working in a collaborative environment, this role is perfect for you. Join us and be part of our dynamic team driving impactful data initiatives.,

As a Senior Big Data Developer specializing in PySpark, with a robust 57 years of experience, you will be responsible for designing, developing, and optimizing scalable data pipelines. Collaboration with cross-functional teams is essential to deliver reliable, high-performance data solutions in the Apache Big Data ecosystem. Your primary responsibilities will encompass building and enhancing large-scale data processing pipelines using PySpark, working with tools like Apache Hadoop, Hive, HDFS, Spark, and more to manage big data workloads efficiently. Additionally, you will be engaged in ETL development, data ingestion, transformation, integration from diverse sources, and collaborating with Data Scientists, Analysts, and business stakeholders to provide data-driven solutions. Ensuring the availability, performance, and scalability of data jobs will be crucial, along with implementing data quality checks, and lineage tracking for auditability. Developing reusable code in Python and Java, contributing to architectural design, performance tuning, tech discussions, and utilizing Apache NiFi for automated data flow management when applicable are integral parts of this role. Your qualifications should include expertise in PySpark, distributed data processing, and a firm grasp of the Apache ecosystem components such as Hadoop, Hive, HDFS, and Spark. Experience with ETL pipelines, data modeling, data warehousing, proficiency in Python, Java, and familiarity with SQL/NoSQL databases are essential. Hands-on experience with Git, CI/CD tools, code versioning, and knowledge of Apache NiFi, real-time data streaming tools, cloud platforms like AWS, Azure, or GCP, Docker, Kubernetes, or other orchestration tools would be advantageous. Your soft skills should encompass excellent problem-solving, analytical abilities, strong collaboration, communication skills, and the ability to work independently and within a team. A Bachelors or Masters degree in Computer Science, Information Systems, or a related field is required. If you are enthusiastic about big data and wish to contribute to impactful projects, please submit your updated resume to nikhil.kumar@krtrimaiq.ai.,

6+ years of experience in information technology, Minimum of 3-5 years of experience in managing and administering Hadoop/Cloudera environments. Cloudera CDP (Cloudera Data Platform), Cloudera Manager, and related tools. Hadoop ecosystem components (HDFS, YARN, Hive, HBase, Spark, Impala, etc.). Linux system administration with experience with scripting languages (Python, Bash, etc.) and configuration management tools (Ansible, Puppet, etc.) Tools like Kerberos, Ranger, Sentry), Docker, Kubernetes, Jenkins Cloudera Certified Administrator for Apache Hadoop (CCAH) or similar certification. Cluster Management, Optimization, Best practice implementation, collaboration and support.

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Optimize PySpark jobs for performance tuning, partitioning, and caching strategies. Design and implement real-time and batch data processing solutions. Integrate data pipelines with Kafka, Delta Lake, Iceberg, or Hudi for streaming and incremental updates. Ensure data security, governance, and compliance with industry best practices. Work with data scientists and analysts to prepare and process large-scale datasets for machine learning models. Collaborate with DevOps teams to deploy, monitor, and scale PySpark jobs using CI/CD pipelines, Kubernetes, and containerization. Perform unit testing and validation to ensure data integrity and reliability. Required Skills & Qualifications: 6+ years of experience in big data processing, ETL, and data engineering. Strong hands-on experience with PySpark (Apache Spark with Python). Expertise in SQL, DataFrame API, and RDD transformations. Experience with big data platforms (Hadoop, Hive, HDFS, Spark SQL). Knowledge of cloud data processing services (AWS Glue, EMR, Databricks, Azure Synapse, GCP Dataflow). Proficiency in writing optimized queries, partitioning, and indexing for performance tuning. Experience with workflow orchestration tools like Airflow, Oozie, or Prefect. Familiarity with containerization and deployment using Docker, Kubernetes, and CI/CD pipelines. Strong understanding of data governance, security, and compliance (GDPR, HIPAA, CCPA, etc.). Excellent problem-solving, debugging, and performance optimization skills.

JOB DESCRIPTION Develop, test, and deploy data processing applications using Apache Spark and Scala. Optimize and tune Spark applications for better performance on large-scale data sets. Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. Design and implement high-performance data processing and analytics solutions. Ensure data integrity, accuracy, and security across all processing tasks. Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies. Implement version control and CI/CD pipelines for Spark applications. Required Skills & Experience: Minimum 8 years of experience in application development. Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing. Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop. Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi. Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data. Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL. Knowledge of data warehousing concepts, dimensional modeling, and data lakes. Ability to troubleshoot and optimize Spark and Cloudera platform performance. Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).

Experience in the Big Data technologies (Hadoop, Spark, Nifi, Impala) 5+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines. High proficiency in Scala/Java and Spark for applied large-scale data processing. Expertise with big data technologies, including Spark, Data Lake, and Hive. Solid understanding of batch and streaming data processing techniques. Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion. Expert-level ability to write complex, optimized SQL queries across extensive data volumes. Experience on HDFS, Nifi, Kafka. Experience on Apache Ozone, Delta Tables, Databricks, Axon(Kafka), Spring Batch, Oracle DB Familiarity with Agile methodologies. Obsession for service observability, instrumentation, monitoring, and alerting. Knowledge or experience in architectural best practices for building data lakes.

As a Data Governance Architect at Tiger Analytics, you will play a crucial role in designing, architecting, deploying, and maintaining big data-based data governance solutions. Your responsibilities will include technical management throughout the project life cycle, collaboration with various teams, exploring new technologies, and leading a team of data governance engineers. You are expected to have a minimum of 10 years of technical experience, with at least 5 years in the Hadoop ecosystem and 3 years in Data Governance Solutions. Hands-on experience with Data Governance Solutions is essential, along with a good understanding of data catalog, business glossary, metadata, data quality, data profiling, and data lineage. Expertise in technologies such as the Hadoop ecosystem (HDFS, Hive, Sqoop, Kafka, ELK Stack), Spark, Scala, Python, core/advanced Java, and relevant AWS/GCP components is required. Familiarity with Databricks, Snowflake, designing/building cloud-computing infrastructure solutions, data lake design, full life cycle of a Hadoop solution, distributed computing, HDFS administration, and configuration management is a plus. At Tiger Analytics, we value diversity and inclusivity. We encourage individuals with varying skill sets and qualities to apply, even if they do not meet all the criteria for the role. We are an equal-opportunity employer, and our diverse culture and values promote growth and development tailored to individual aspirations. Your designation and compensation will be determined based on your expertise and experience. We offer competitive compensation packages and additional benefits such as health insurance, virtual wellness platforms, car lease programs, and opportunities to engage with knowledge communities. Join us at Tiger Analytics to be part of a dynamic team that is dedicated to pushing the boundaries of AI and analytics to create real outcomes for businesses.,