7 Apache Hadoop Jobs

RSM is looking for an experienced Hands-On Technical Manager with expertise in big data technologies and multi-cloud platforms to lead our technical team for the financial services industry. The ideal candidate will possess a strong background in big data architecture, cloud computing, and a deep understanding of the financial services industry. As a Technical Manager, you will be responsible for leading technical projects, hands-on development, delivery management, sales and ensuring the successful implementation of data solutions across multiple cloud platforms. This role requires a unique blend of technical proficiency, sales acumen, and presales experience to drive business growth and deliver innovative data solutions to our clients. Responsibilities: Provide technical expertise and guidance on the selection, and hands-on implementation, and optimization of big data platforms, tools, and technologies across multiple cloud environments (e.g., AWS, Azure, GCP, Snowflake, etc.) Architect and build scalable and secure data pipelines, data lakes, and data warehouses to support the storage, processing, and analysis of large volumes of structured and unstructured data. Lead and mentor a team of technical professionals in the design, development, and implementation of big data solutions and data analytics projects within the financial services domain. Stay abreast of emerging trends, technologies, and industry developments in big data, cloud computing, and financial services, and assess their potential impact on the organization. Develop and maintain best practices, standards, and guidelines for data management, data governance, and data security in alignment with regulatory requirements and industry standards. Collaborate with the sales and business development teams to identify customer needs, develop solution proposals, and present technical demonstrations and presentations to prospective clients. Collaborate with cross-functional teams including data scientists, engineers, business analysts, and stakeholders to define project requirements, objectives, and timelines. Basic Qualifications: Bachelor's degree or higher in Computer Science, Information Technology, Business Administration, Engineering or related field. Minimum of ten years of overall technical experience in solution architecture, design, hands-on development with a focus on big data technologies, multi-cloud platforms, and with at-least 5 years of experience specifically in financial services. Strong understanding of the financial services industry capital markets, retail and business banking, asset management, insurance, etc. In-depth knowledge of big data technologies such as Hadoop, Spark, Kafka, and cloud platforms such as AWS, Azure, GCP, Snowflake, Databricks, etc. Experience with SQL, Python, Pyspark or other programming languages used for data transformation, analysis, and automation. Excellent communication, presentation, and interpersonal skills, with the ability to articulate technical concepts to both technical and non-technical audiences. Hands-on experience extracting (ETL using CDC, Transaction Logs, Incremental) and processing large data sets for Streaming and Batch data loads. Ability to work from our Hyderabad, India office at least twice a week Preferred Qualifications: Professional certifications in cloud computing (e.g., AWS Certified Solutions Architect, Microsoft Certified Azure Solutions Architect, Azure Data Engineer, SnowPro Core) and/or big data technologies. Experience with Power BI, Tableau or other Reporting and Data Visualization tools Familiarity with DevOps practices, CI/CD pipelines, and infrastructure-as-code tools Education/Experience: Bachelor's degree in MIS, CS, Engineering or equivalent field. Master's degree is CS or MBA is preferred. Advanced Data and Cloud Certifications are a plus.

Job Description AWS Data engineer Hadoop Migration We are seeking an experienced AWS Principal Data Architect to lead the migration of Hadoop DWH workloads from on-premise to AWS EMR. As an AWS Data Architect, you will be a recognized expert in cloud data engineering, developing solutions designed for effective data processing and warehousing requirements of large enterprises. You will be responsible for designing, implementing, and optimizing the data architecture in AWS, ensuring highly scalable, flexible, secured and resilient cloud architectures solving business problems and helps accelerate the adoption of our clients data initiatives on the cloud. Key Responsibilities: Lead the migration of Hadoop workloads from on-premise to AWS-EMR stack. Design and implement data architectures on AWS, including data pipelines, storage, and security. Collaborate with cross-functional teams to ensure seamless migration and integration. Optimize data architectures for scalability, performance, and cost-effectiveness. Develop and maintain technical documentation and standards. Provide technical leadership and mentorship to junior team members. Work closely with stakeholders to understand business requirements, and ensure data architectures meet business needs. Work alongside customers to build enterprise data platforms using AWS data services like Elastic Map Reduce (EMR), Redshift, Kinesis, Data Exchange, Data Sync, RDS , Data Store, Amazon MSK, DMS, Glue, Appflow, AWA Zero-ETL, Glue Data Catalog, Athena, Lake Formation, S3, RMS, Data Zone, Amazon MWAA, APIs Kong Deep understanding of Hadoop components, conceptual processes and system functioning and relative components in AWS EMR and other AWS services. Good experience on Spark-EMR Experience in Snowflake/Redshift Good idea of AWS system engineering aspects of setting up CI-CD pipelines on AWS using Cloudwatch, Cloudtrail, KMS, IAM IDC, Secret Manager, etc Extract best-practice knowledge, reference architectures, and patterns from these engagements for sharing with the worldwide AWS solution architect community Basic Qualifications: 10+ years of IT experience with 5+ years of experience in Data Engineering and 5+ years of hands-on experience in AWS Data/EMR Services (e.g. S3, Glue, Glue Catalog, Lake Formation) Strong understanding of Hadoop architecture, including HDFS, YARN, MapReduce, Hive, HBase. Experience with data migration tools like Glue, Data Sync. Excellent knowledge of data modeling, data warehousing, ETL processes, and other Data management systems. Strong understanding of security and compliance requirements in cloud. Experience in Agile development methodologies and version control systems. Excellent communication an leadership skills. Ability to work effectively across internal and external organizations and virtual teams. Deep experience on AWS native data services including Glue, Glue Catalog, EMR, Spark-EMR, Data Sync, RDS, Data Exchange, Lake Formation, Athena, AWS Certified Data Analytics – Specialty. AWS Certified Solutions Architect – Professional. Experience on Containerization and serverless computing. Familiarity with DevOps practices and automation tools. Experience in Snowflake/Redshift implementation is additionally Azure Data Engineer job description As an Azure Data Engineer, the candidate is expected to be specializing in designing, implementing, and managing large-scale data solutions on the Microsoft Azure cloud platform. They possess expertise in various aspects of data engineering like data storage, data integrations, analytics etc using Azure data services. Ideal candidate will have a strong background in data engineering, Azure cloud services and data processing technologies. # Requirements: 1. 5+ years of experience in data engineering, preferably on Microsoft Azure. This will increase with senior positions as other DE roles we have. 2. Strong knowledge of Azure cloud services, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Storage. 3. Experience with data processing technologies, such as Apache Spark, Apache Hadoop, and SQL. 4. Strong understanding of data modeling, data warehousing, and data governance. 5. Experience with data security, compliance, and regulatory requirements. 6. Strong programming skills in languages such as Python, Scala, or Java. 7. Experience with agile development methodologies and version control systems such as Git 8. Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect expert. 9. Experience with containerization technologies such as Docker. 10. Knowledge of data visualization tools, such as Power BI, Tableau etc 11. Experience in machine learning and AI technologies will be added advantage. Soft Skills: Problem-solving: Solve complex data problems effectively. Communication: Communicate clearly with both technical and non-technical team members. Attention to detail: Pay close attention to detail to ensure accuracy. Adaptability and learning: Stay up to date with new technologies and trends. Teamwork: Collaborate well with others for project success. Leadership and mentorship: Take on leadership roles and mentor junior team members for growth.

Azure Data Engineer job description As an Azure Data Engineer, the candidate is expected to be specializing in designing, implementing, and managing large-scale data solutions on the Microsoft Azure cloud platform. They possess expertise in various aspects of data engineering like data storage, data integrations, analytics etc using Azure data services. Ideal candidate will have a strong background in data engineering, Azure cloud services and data processing technologies. # Requirements: 1. 5+ years of experience in data engineering, preferably on Microsoft Azure. This will increase with senior positions as other DE roles we have. 2. Strong knowledge of Azure cloud services, including Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Storage. 3. Experience with data processing technologies, such as Apache Spark, Apache Hadoop, and SQL. 4. Strong understanding of data modeling, data warehousing, and data governance. 5. Experience with data security, compliance, and regulatory requirements. 6. Strong programming skills in languages such as Python, Scala, or Java. 7. Experience with agile development methodologies and version control systems such as Git 8. Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect expert. 9. Experience with containerization technologies such as Docker. 10. Knowledge of data visualization tools, such as Power BI, Tableau etc 11. Experience in machine learning and AI technologies will be added advantage. Soft Skills: Problem-solving: Solve complex data problems effectively. Communication: Communicate clearly with both technical and non-technical team members. Attention to detail: Pay close attention to detail to ensure accuracy. Adaptability and learning: Stay up to date with new technologies and trends. Teamwork: Collaborate well with others for project success. Leadership and mentorship: Take on leadership roles and mentor junior team members for growth.

Data engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets. Job Description - Grade Specific The involves leading and managing a team of data engineers, overseeing data engineering projects, ensuring technical excellence, and fostering collaboration with stakeholders. They play a critical role in driving the success of data engineering initiatives and ensuring the delivery of reliable and high-quality data solutions to support the organization's data-driven objectives. Skills (competencies) Ab Initio Agile (Software Development Framework) Apache Hadoop AWS Airflow AWS Athena AWS Code Pipeline AWS EFS AWS EMR AWS Redshift AWS S3 Azure ADLS Gen2 Azure Data Factory Azure Data Lake Storage Azure Databricks Azure Event Hub Azure Stream Analytics Azure Sunapse Bitbucket Change Management Client Centricity Collaboration Continuous Integration and Continuous Delivery (CI/CD) Data Architecture Patterns Data Format Analysis Data Governance Data Modeling Data Validation Data Vault Modeling Database Schema Design Decision-Making DevOps Dimensional Modeling GCP Big Table GCP BigQuery GCP Cloud Storage GCP DataFlow GCP DataProc Git Google Big Tabel Google Data Proc Greenplum HQL IBM Data Stage IBM DB2 Industry Standard Data Modeling (FSLDM) Industry Standard Data Modeling (IBM FSDM)) Influencing Informatica IICS Inmon methodology JavaScript Jenkins Kimball Linux - Redhat Negotiation Netezza NewSQL Oracle Exadata Performance Tuning Perl Platform Update Management Project Management PySpark Python R RDD Optimization SantOs SaS Scala Spark Shell Script Snowflake SPARK SPARK Code Optimization SQL Stakeholder Management Sun Solaris Synapse Talend Teradata Time Management Ubuntu Vendor Management

Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : Apache Hadoop Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years of fulltime education Summary :As an Application Lead for Packaged Application Development, you will be responsible for leading the effort to design, build, and configure applications using Apache Hadoop. Your typical day will involve collaborating with cross-functional teams, managing project timelines, and ensuring the successful delivery of high-quality applications. Roles & Responsibilities: Lead the design, development, and implementation of applications using Apache Hadoop, acting as the primary point of contact for the project. Collaborate with cross-functional teams, including business analysts, developers, and testers, to ensure the successful delivery of high-quality applications. Manage project timelines, ensuring that deliverables are completed on time and within budget. Provide technical guidance and mentorship to junior team members, ensuring that best practices are followed and that team members are continuously learning and growing. Stay up-to-date with the latest advancements in Apache Hadoop and related technologies, integrating innovative approaches for sustained competitive advantage. Professional & Technical Skills: The candidate should have a minimum of 7 years of experience in Apache Hadoop. The ideal candidate will possess a strong educational background in computer science or a related field, along with a proven track record of delivering impactful data-driven solutions. Must To Have Skills:Strong experience in Apache Hadoop. Good To Have Skills:Experience with related technologies such as Apache Spark, Hive, and HBase. Solid understanding of distributed computing principles. Experience with Java or Scala programming languages. Experience with data warehousing and ETL processes. Strong problem-solving and analytical skills. Additional Information: This position is based at our Pune office.

• Lead advanced R&D initiatives • Design innovative machine learning algorithms • Analyze large datasets to solve complex problems • Lead data asset evaluation & acquisition • Develop data-driven models & validate performance

Data engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets. Job Description - Grade Specific The involves leading and managing a team of data engineers, overseeing data engineering projects, ensuring technical excellence, and fostering collaboration with stakeholders. They play a critical role in driving the success of data engineering initiatives and ensuring the delivery of reliable and high-quality data solutions to support the organization's data-driven objectives. Skills (competencies) Ab Initio Agile (Software Development Framework) Apache Hadoop AWS Airflow AWS Athena AWS Code Pipeline AWS EFS AWS EMR AWS Redshift AWS S3 Azure ADLS Gen2 Azure Data Factory Azure Data Lake Storage Azure Databricks Azure Event Hub Azure Stream Analytics Azure Sunapse Bitbucket Change Management Client Centricity Collaboration Continuous Integration and Continuous Delivery (CI/CD) Data Architecture Patterns Data Format Analysis Data Governance Data Modeling Data Validation Data Vault Modeling Database Schema Design Decision-Making DevOps Dimensional Modeling GCP Big Table GCP BigQuery GCP Cloud Storage GCP DataFlow GCP DataProc Git Google Big Tabel Google Data Proc Greenplum HQL IBM Data Stage IBM DB2 Industry Standard Data Modeling (FSLDM) Industry Standard Data Modeling (IBM FSDM)) Influencing Informatica IICS Inmon methodology JavaScript Jenkins Kimball Linux - Redhat Negotiation Netezza NewSQL Oracle Exadata Performance Tuning Perl Platform Update Management Project Management PySpark Python R RDD Optimization SantOs SaS Scala Spark Shell Script Snowflake SPARK SPARK Code Optimization SQL Stakeholder Management Sun Solaris Synapse Talend Teradata Time Management Ubuntu Vendor Management