185 Aws Emr Jobs - Page 2

Urgent Hiring: AWS Data Engineer, Senior Data Engineers & Lead Data Engineers Apply Now: Send your resume to heena.ruchwani@gspann.com Location: Bangalore (5+ Years Experience) Company: GSPANN Technologies, Inc. GSPANN Technologies is seeking talented professionals with 5+ years of experience to join our team in Bangalore. We are looking for immediate joiners who are passionate about data engineering and eager to take on exciting challenges. Key Skills & Experience: 5+ years of hands-on experience with AWS Data Services (Glue, Redshift, S3, Lambda, EMR, Athena, etc.) Strong expertise in Big Data Technologies (Spark, Hadoop, Kafka) Proficiency in SQL, Python, and Scala Hands-on experience with ETL pipelines, data modeling, and cloud-based data solutions Location: Bangalore Apply Now: Send your resume to heena.ruchwani@gspann.com Immediate Joiners Preferred! If you're ready to contribute to dynamic, data-driven projects and advance your career with GSPANN Technologies, apply today!

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala ; Minimum 3 years of experience on Cloud Data Platforms on AWS; Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Exposure to streaming solutions and message brokers like Kafka technologies Preferred technical and professional experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs.Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Developed the Pysprk code for AWS Glue jobs and for EMR. Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution.. Developed Python and pyspark programs for data analysis. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine). Developed Hadoop streaming Jobs using python for integrating python API supported applications.. Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations.. Re- write some Hive queries to Spark SQL to reduce the overall batch time Preferred technical and professional experience Understanding of Devops. Experience in building scalable end-to-end data ingestion and processing solutions Experience with object-oriented and/or functional programming languages, such as Python, Java and Scala

Develop full-stack forecasting and simulation tools using React, Node.js, and Python on AWS Modernize Excel/VBA tools into scalable cloud-native web applications Deploy solutions efficiently with AWS services, CI/CD, and infrastructure-as-code tools Food allowance Annual bonus Health insurance Provident fund

Why Join Iris Are you ready to do the best work of your career at one of India's Top 25 Best Workplaces in IT industry Do you want to grow in an award-winning culture that truly values your talent and ambitions Join Iris Software - one of the fastest-growing IT services companies - where you own and shape your success story . About Us At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services. Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation. Working with Us At Iris, every role is more than a job - it's a launchpad for growth. Our Employee Value Proposition, Build Your Future. Own Your Journey. reflects our belief that people thrive when they have ownership of their career and the right opportunities to shape it. We foster a culture where your potential is valued, your voice matters, and your work creates real impact. With cutting-edge projects, personalized career development, continuous learning and mentorship, we support you to grow and become your best - both personally and professionally. Curious what it's like to work at Iris Head to this video for an inside look at the people, the passion, and the possibilities. Watch it . Job Description Design and implement scalable data processing solutions usingApache SparkandJava. Develop and maintain high-performance backend services and APIs. Collaborate with data scientists, analysts, and other engineers to understand data requirements. Optimize Spark jobs for performance and cost-efficiency. Ensure code quality through unit testing, integration testing, and code reviews. Work with large-scale datasets in distributed environments (e.g., Hadoop, AWS EMR, Databricks). Monitor and troubleshoot production systems and pipelines. Experience in Agile Development Process. Experience in leading a 3-5 member team on the technology front Excellent communication skills, problem solving and debugging and troubleshooting Skills. Mandatory Competencies Programming Language - Java - Core Java (java 8+) Architecture - Architectural Patterns - Microservices Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark Tech - Unit Testing Data Science and Machine Learning - Data Science and Machine Learning - Databricks Big Data - Big Data - Hadoop Cloud - AWS - Tensorflow on AWS, AWS Glue, AWS EMR, Amazon Data Pipeline, AWS Redshift Agile - Agile - Extreme Programming Big Data - Big Data - SPARK Beh - Communication and collaboration Perks and Benefits for Irisians Iris provides world-class benefits for a personalized employee experience. These benefits are designed to support financial, health and well-being needs of Irisians for a holistic professional and personal growth. Click to view the benefits.

Why Join Iris Are you ready to do the best work of your career at one of India's Top 25 Best Workplaces in IT industry Do you want to grow in an award-winning culture that truly values your talent and ambitions Join Iris Software - one of the fastest-growing IT services companies - where you own and shape your success story . About Us At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services. Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation. Working with Us At Iris, every role is more than a job - it's a launchpad for growth. Our Employee Value Proposition, Build Your Future. Own Your Journey. reflects our belief that people thrive when they have ownership of their career and the right opportunities to shape it. We foster a culture where your potential is valued, your voice matters, and your work creates real impact. With cutting-edge projects, personalized career development, continuous learning and mentorship, we support you to grow and become your best - both personally and professionally. Curious what it's like to work at Iris Head to this video for an inside look at the people, the passion, and the possibilities. Watch it . Job Description Design and implement scalable data processing solutions usingApache SparkandJava. Develop and maintain high-performance backend services and APIs. Collaborate with data scientists, analysts, and other engineers to understand data requirements. Optimize Spark jobs for performance and cost-efficiency. Ensure code quality through unit testing, integration testing, and code reviews. Work with large-scale datasets in distributed environments (e.g., Hadoop, AWS EMR, Databricks). Monitor and troubleshoot production systems and pipelines. Experience in Agile Development Process. Experience in leading a 3-5 member team on the technology front Excellent communication skills, problem solving and debugging and troubleshooting Skills. Mandatory Competencies Programming Language - Java - Core Java (java 8+) Architecture - Architectural Patterns - Microservices Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark Tech - Unit Testing Data Science and Machine Learning - Data Science and Machine Learning - Databricks Big Data - Big Data - Hadoop Cloud - AWS - Tensorflow on AWS, AWS Glue, AWS EMR, Amazon Data Pipeline, AWS Redshift Agile - Agile - Extreme Programming Big Data - Big Data - SPARK Beh - Communication and collaboration Perks and Benefits for Irisians Iris provides world-class benefits for a personalized employee experience. These benefits are designed to support financial, health and well-being needs of Irisians for a holistic professional and personal growth. Click to view the benefits.

Notice Period : Immediate - 30 days Mandatory Skills : Big Data, Python, SQL, Spark/Pyspark, AWS Cloud JD and required Skills & Responsibilities : - Actively participate in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing, roll-out, and support. - Solve complex business problems by utilizing a disciplined development methodology. - Produce scalable, flexible, efficient, and supportable solutions using appropriate technologies. - Analyse the source and target system data. Map the transformation that meets the requirements. - Interact with the client and onsite coordinators during different phases of a project. - Design and implement product features in collaboration with business and Technology stakeholders. - Anticipate, identify, and solve issues concerning data management to improve data quality. - Clean, prepare, and optimize data at scale for ingestion and consumption. - Support the implementation of new data management projects and re-structure the current data architecture. - Implement automated workflows and routines using workflow scheduling tools. - Understand and use continuous integration, test-driven development, and production deployment frameworks. - Participate in design, code, test plans, and dataset implementation performed by other data engineers in support of maintaining data engineering standards. - Analyze and profile data for the purpose of designing scalable solutions. - Troubleshoot straightforward data issues and perform root cause analysis to proactively resolve product issues. Required Skills : - 5+ years of relevant experience developing Data and analytic solutions. - Experience building data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive & PySpark - Experience with relational SQL. - Experience with scripting languages such as Python. - Experience with source control tools such as GitHub and related dev process. - Experience with workflow scheduling tools such as Airflow. - In-depth knowledge of AWS Cloud (S3, EMR, Databricks) - Has a passion for data solutions. - Has a strong problem-solving and analytical mindset - Working experience in the design, Development, and test of data pipelines. - Experience working with Agile Teams. - Able to influence and communicate effectively, both verbally and in writing, with team members and business stakeholders - Able to quickly pick up new programming languages, technologies, and frameworks. - Bachelor's degree in computer science

Educational Requirements Bachelor of Engineering Service Line Data & Analytics Unit Responsibilities A day in the life of an Infoscion As part of the Infosys delivery team, your primary role would be to interface with the client for quality assurance, issue resolution and ensuring high customer satisfaction. You will understand requirements, create and review designs, validate the architecture and ensure high levels of service offerings to clients in the technology domain. You will participate in project estimation, provide inputs for solution delivery, conduct technical risk planning, perform code reviews and unit test plan reviews. You will lead and guide your teams towards developing optimized high quality code deliverables, continual knowledge management and adherence to the organizational guidelines and processes. You would be a key contributor to building efficient programs/ systems and if you think you fit right in to help our clients navigate their next in their digital transformation journey, this is the place for you!If you think you fit right in to help our clients navigate their next in their digital transformation journey, this is the place for you! Technical and Professional Requirements: Primary skills: Technology->Big Data - Data Processing->Map Reduce Preferred Skills: Technology->Big Data - Data Processing->Map Reduce

As a Data Engineer, you will be responsible for building scalable data pipelines using PySpark. Your role will involve implementing complex business logic using Spark SQL, DataFrame, and RDD APIs. You should have strong programming skills in Python, with a solid understanding of data structures, algorithms, and software engineering principles. Your expertise in designing, developing, and maintaining batch and streaming data pipelines will be crucial. You should be familiar with ETL/ELT processes and best practices for data transformation, data quality, and performance optimization. Knowledge of the modern data engineering ecosystem, including distributed data processing, storage systems, and workflow orchestration tools like Apache Airflow, dbt, and Delta Lake, is desirable. Experience with cloud data platforms, preferably AWS, is preferred. You should have hands-on experience with AWS services such as S3 for data lake, Glue/EMR for Spark workloads, Lambda, Step Functions for orchestration, and Redshift or other cloud data warehouses. As an expert in Spark APIs, you should be able to choose and apply the right APIs (DataFrame, Dataset, RDD) for efficient implementation of business logic at scale. This role offers a 12+ month contract with a likely long-term opportunity, following a hybrid work mode with an immediate to 15 days notice period. If you have a passion for data engineering and the skills mentioned above, we would like to hear from you.,

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Total 3-5+ years of experience in Data Management (DW, DL, Data Platform, Lakehouse) and Data Engineering skills Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala ; Minimum 3 years of experience on Cloud Data Platforms on AWS; Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Preferred technical and professional experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc. Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala ; Minimum 3 years of experience on Cloud Data Platforms on AWS; Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Exposure to streaming solutions and message brokers like Kafka technologies. Preferred technical and professional experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers

We are Hiring: AWS Data Architect at Coforge Ltd. Join Coforge Ltd as a Lead AWS Data Architect. Job Location: Hyderabad (Onsite Only) Experience Required: 1215 Years Position Type: Full-Time Company: Coforge Ltd. How to Apply Interested candidates can share their CV directly with: Gaurav.2.Kumar@coforge.com WhatsApp: 9667427662 About the Role:- Coforge Ltd is seeking a visionary Lead AWS Data Architect to spearhead our cloud-first data initiatives. This is a strategic leadership role where youll shape the future of data architecture, mentor top-tier talent, and deliver scalable, secure, and innovative solutions using cutting-edge AWS technologies. Key Responsibilities:- Team Leadership & Mentorship:- Guide and grow a team of skilled data engineers and architects, fostering a culture of excellence and innovation. Cloud Data Architecture:- Design and implement robust, scalable data pipelines using Python , Airflow , and AWS services like S3 , Glue , and EMR . Real-Time Data Streaming:- Architect real-time data solutions using Kafka , Amazon SQS , and EventBridge to enable responsive and intelligent systems. System Integration:- Seamlessly connect diverse systems using AppFlow , REST APIs , and other integration tools. Data Warehousing & Modeling:- Build optimized data warehouses with strong dimensional modeling practices to support analytics and reporting. Governance & Security:- Ensure all solutions comply with enterprise data governance, privacy, and security standards. What Were Looking For:- Experience:- 1015 years in data engineering and architecture. Minimum 3 years in a technical leadership role. Technical Skills:- Expert in Python , Airflow , and AWS ecosystem. Hands-on experience with Kafka , SQS , EventBridge. Strong understanding of data warehousing , ETL/ELT and API integrations. Education:- Bachelors or masters degree in computer science, Engineering, or a related field. Soft Skills:- Excellent communication, stakeholder management, and problem-solving abilities. Passion for mentoring and driving team success. Why Coforge? At Coforge, we’re not just building systems—we’re transforming industries. Join a team that values innovation, collaboration, and continuous learning. Be part of a data-driven revolution. About Coforge Ltd Coforge Ltd is a globally recognized digital services and solutions provider, headquartered in Noida, Uttar Pradesh, India . Formerly known as NIIT Technologies , the company rebranded to Coforge in August 2020 , marking a strategic shift toward deeper specialization and innovation in digital transformation. With over 40 years of industry experience , Coforge operates in more than 21 countries , including the United States, United Kingdom, Australia, Singapore, and across Europe and Asia-Pacific. It maintains 30 global delivery centers and employs a workforce of over 32,000 professionals , delivering high-impact solutions across industries such as: Banking & Financial Services Insurance Travel & Transportation Healthcare Manufacturing & Distribution Media & Government Coforge’s core service offerings include: Digital Engineering & Application Development Cloud Infrastructure & Automation Artificial Intelligence & Data Management Cybersecurity & Digital Process Automation Enterprise Applications including SAP Business Process Services (BPS) The company is known for its product engineering approach , leveraging proprietary platforms and emerging technologies like Generative AI , Cloud , and Data Integration to help clients become intelligent, high-growth enterprises.

As a Data Engineer, you will be responsible for designing, developing, and optimizing data pipelines and ETL workflows using AWS Glue, AWS Lambda, and Apache Spark. Your role will involve implementing big data processing solutions utilizing AWS EMR and AWS Redshift. You will also be tasked with developing and maintaining data lakes and data warehouses on AWS, including S3, Redshift, and RDS. Ensuring data quality, integrity, and governance will be a key aspect of your responsibilities, which will be achieved through leveraging AWS Glue Data Catalog and AWS Lake Formation. It will be essential for you to optimize data storage and processing for both performance and cost efficiency. Working with structured, semi-structured, and unstructured data across various storage formats such as Parquet, Avro, and JSON will be part of your daily tasks. Automation and orchestration of data workflows using AWS Step Functions and Apache Airflow will also fall within your scope of work. You will be expected to implement best practices for CI/CD pipelines in data engineering with AWS CodePipeline and AWS CodeBuild. Monitoring, troubleshooting, and optimizing data pipeline performance and scalability will be critical to ensuring smooth operations. Collaborating with cross-functional teams, including data scientists, analysts, and software engineers, will be necessary to drive successful outcomes. Your role will require a minimum of 6 years of experience in data engineering and big data processing. Proficiency in AWS cloud services like AWS Glue, AWS Lambda, AWS Redshift, AWS EMR, and S3 is paramount. Strong skills in Python for data engineering tasks, hands-on experience with Apache Spark and SQL, as well as knowledge of data modeling, schema design, and performance tuning are essential. Understanding AWS Lake Formation and Lakehouse principles, experience with version control using Git, and familiarity with CI/CD pipelines are also required. Knowledge of data security, compliance, and governance best practices is crucial. Experience with real-time streaming technologies such as Kafka and Kinesis will be an added advantage. Strong problem-solving, analytical, and communication skills are key attributes for success in this role.,

About the Opportunity : Join a dynamic leader in the cloud data engineering sector, specializing in advanced data solutions and real-time analytics for enterprise clients. This role offers an on-site opportunity in India to work on cutting-edge AWS infrastructures where innovation is at the forefront of business transformation. This opportunity is ideal for professionals with 4+ years of proven experience in AWS data engineering, Python, and PySpark. You will contribute to designing, optimizing, and maintaining scalable data pipelines that drive business intelligence and operational efficiency. Role & Responsibilities : - Design, develop, and maintain robust AWS-based data pipelines using Python and PySpark. - Implement efficient ETL processes, ensuring data integrity and optimal performance across AWS services (S3, Glue, EMR, Redshift). - Collaborate with cross-functional teams to integrate data engineering solutions within broader business-critical applications. - Troubleshoot and optimize existing data workflows, ensuring high availability, scalability, and security of cloud solutions. - Exercise best practices in coding, version control, and documentation to maintain a high standard of engineering excellence. Skills & Qualifications : Must-Have : 4+ years of hands-on experience in AWS data engineering with proven expertise in Python and PySpark. Must-Have : Proficiency in developing and maintaining ETL processes, using AWS services such as S3, Glue, EMR, and Redshift. Must-Have : Strong problem-solving skills and a deep understanding of data modeling, data warehousing concepts, and performance optimization. Preferred : Experience with AWS Lambda, Airflow, or similar cloud orchestration tools. Preferred : Familiarity with containerization, CI/CD pipelines, and infrastructure-as-code (e.g., CloudFormation, Terraform). Preferred : AWS certifications or equivalent cloud credentials. Benefits & Culture Highlights : - Work in a collaborative, fast-paced environment that rewards innovation and continuous improvement. - Enjoy opportunities for professional growth and skill development through ongoing projects and training. - Benefit from competitive compensation and the ability to work on transformative cloud technology solutions.

We are looking for a Manager Software Engineering to join our IMS Team in Bangalore . This is an amazing opportunity to work on Big Data technologies involved in content ingestion. The team consists of 10-12 engineers and is reporting to the Sr Manager. We have a great skill set in Spark, Java, Scala, Hive, Sql, XSLT, AWS EMR, S3, etc and we would love to speak with you if you have skills in the same. About You experience, education, skills, and accomplishments: Work Experience: Minimum 7 years experience in Big Data projects involved in content ingestion, curation, transformation Technical Skill: Databricks, Spark, Python/Java , Scala, AWS EMR, S3, SQS, Hive, XSLT, Education (bachelors degree in computer science or related degree or at least 4 years of equivalent relevant experience) It would be great if you also had: Experience in analyzing and optimizing performance Exposure to any automation test frameworks Java programming What will you be doing in this role? Lead customer solutions by understanding client needs, proposing tailored solutions, and overseeing their implementation. Drive innovation by staying current of industry trends, identifying opportunities for improvement, and implementing creative solutions. Direct design solutions by working closely with design teams to develop innovative and user-centric products. Regularly evaluate team performance, assess technical capabilities, provide constructive feedback to colleagues, and escalate issues to leadership when necessary. Foster interdepartmental communication by facilitating collaboration between Solutions Engineering, DevOps, Data Science, and other relevant teams. Deliver quarterly roadmaps to align with business goals and objectives. Estimate completion time for development projects in a quarter, ensuring timely delivery of solutions. Manage a team of 4 to 5 colleagues. Lead, direct and review the teams work to achieve operational or functional targets or objectives with measurable contribution towards the achievement of immediate and short-term results for the team and department. Establish and communicate team goals that support and advance team and department objectives; establish metrics or other performance measures to track progress. Understand and explain policies, practices and procedures within immediate area of responsibility to stakeholders within and outside of the team. Deliver consistent and timely training, guidance and feedback to encourage and promote colleague success and growth through regular developmental conversations and identification of developmental opportunities. About the Team: We are a 12 member strong team based out of India. We have end to end development and support ownership of the IMS product that plays a vital role in content ingestion, aggregation, transformation, and enrichment of content, that is transmitted to downstream applications. Our development methodology is Agile , and our system heavily relies on AWS EMR services.

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala ; Minimum 3 years of experience on Cloud Data Platforms on AWS; Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Exposure to streaming solutions and message brokers like Kafka technologies Preferred technical and professional experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers

The Machine Learning Engineer position based in GGN requires a professional with 6-9 years of experience. The ideal candidate should possess expertise in Spark, SQL, Python/Scala, AWS EMR, AWS S3, ML Life Cycle Management, and Machine Learning Operations (ML Ops). Additionally, experience with Airflow or any other orchestrator is considered a good to have qualification. Experience with Kafka, Spark Streaming, Datadog, and Kubernetes are also valued assets for this role. If you meet these qualifications and are passionate about machine learning, this position could be an excellent fit for you.,

As a Python Developer joining our team in partnership with a renowned financial institution, you will be responsible for analyzing existing SAS DI pipelines and SQL-based transformations. Your role will involve translating and optimizing SAS SQL logic into Python code using frameworks like Pyspark. You will play a crucial part in developing and maintaining scalable ETL pipelines using Python on AWS EMR, implementing data transformation, cleansing, and aggregation logic to meet business requirements. Your expertise will be utilized to design modular and reusable code for distributed data processing tasks on EMR clusters. You will also be integrating EMR jobs with upstream and downstream systems, including AWS S3, Snowflake, and Tableau, as well as developing Tableau reports for business reporting purposes. To excel in this role, the following skills are mandatory: - A minimum of 6 years of experience in ETL development, with at least 5 years of hands-on experience with AWS EMR. - A Bachelor's degree in Computer Science, Data Science, Statistics, or a related field. - Proficiency in Python for data processing and scripting. - Strong SQL skills and experience with one or more ETL tools such as SAS DI or Informatica. - Hands-on experience with AWS services such as EMR, S3, IAM, VPC, and Glue. - Familiarity with data storage systems like Snowflake or RDS. - Excellent communication skills and the ability to work collaboratively in a team environment. - Strong problem-solving skills and the capability to work independently.,

We are searching for a highly skilled and seasoned Senior ETL & Data Streaming Engineer with over 10 years of experience to take on a crucial role in the design, development, and maintenance of our robust data pipelines. The ideal candidate will possess in-depth expertise in batch ETL processes as well as real-time data streaming technologies, along with extensive hands-on experience with AWS data services. A proven track record of working with Data Lake architectures and traditional Data Warehousing environments is a must. Your responsibilities will include designing, developing, and implementing highly scalable, fault-tolerant, and performant ETL processes using leading ETL tools to extract, transform, and load data from diverse source systems into our Data Lake and Data Warehouse. You will also be tasked with architecting and constructing batch and real-time data streaming solutions using technologies like Talend, Informatica, Apache Kafka, or AWS Kinesis to facilitate immediate data ingestion and processing requirements. Furthermore, you will need to leverage and optimize various AWS data services such as AWS S3, AWS Glue, AWS Redshift, AWS Lake Formation, AWS EMR, and others to develop and manage data pipelines. Collaboration with data architects, data scientists, and business stakeholders to comprehend data requirements and translate them into efficient data pipeline solutions is a key aspect of the role. It will also be essential for you to ensure data quality, integrity, and security across all data pipelines and storage solutions, as well as monitor, troubleshoot, and optimize existing data pipelines for performance, cost-efficiency, and reliability. Additionally, you will be responsible for developing and maintaining comprehensive documentation for all ETL and streaming processes, data flows, and architectural designs, and implementing data governance policies and best practices within the Data Lake and Data Warehouse environments. As a mentor to junior engineers, you will contribute to fostering a culture of technical excellence and continuous improvement. Staying updated on emerging technologies and industry best practices in data engineering, ETL, and streaming will also be expected. Required Qualifications: - 10+ years of progressive experience in data engineering, focusing on ETL, ELT, and data pipeline development. - Extensive hands-on experience with commercial or open-source ETL tools (Talend). - Proven experience with real-time data ingestion and processing using platforms such as AWS Glue, Apache Kafka, AWS Kinesis, or similar. - Proficiency with AWS S3, AWS Glue, AWS Redshift, AWS Lake Formation, and potentially AWS EMR. - Strong background in traditional data warehousing concepts, dimensional modeling, and DWH design principles. - Proficient in SQL and at least one scripting language (e.g., Python, Scala) for data manipulation and automation. - Strong understanding of relational databases and NoSQL databases. - Experience with version control systems (e.g., Git). - Excellent analytical and problem-solving skills with attention to detail. - Strong verbal and written communication skills for conveying complex technical concepts to diverse audiences. Preferred Qualifications: - Certifications in AWS Data Analytics or related areas.,

You are a skilled and proactive Python / PySpark Developer sought to join the data engineering or analytics team. Your responsibilities will include building scalable data pipelines, performing large-scale data processing, and collaborating with data scientists, analysts, and business stakeholders. You will design, develop, and optimize ETL data pipelines using PySpark on big data platforms (e.g., Hadoop, Databricks, EMR). Writing clean, efficient, and modular code in Python for data processing and integration tasks is essential. Working with large datasets to extract insights, transform raw data, and ensure data quality will be part of your daily tasks. Collaborating with cross-functional teams to understand business requirements and translating them into technical solutions is crucial. You will also need to implement performance tuning and debugging of PySpark jobs, monitor and troubleshoot data workflows and batch jobs in production environments, and document solutions while maintaining code repositories (e.g., Git). Required qualifications include proficiency in Python with experience in building data-centric applications, strong experience with PySpark, understanding of Spark internals, hands-on experience with Hadoop ecosystem, Hive, or cloud-based big data platforms like AWS EMR, Azure Databricks, or GCP DataProc, familiarity with workflow orchestration tools, good understanding of SQL and relational databases, experience with version control systems, strong problem-solving skills, and a Bachelor's degree in Computer Science, Engineering, or a related field. Preferred qualifications include experience with CI/CD pipelines and DevOps practices, knowledge of data warehousing and data modeling, exposure to streaming technologies like Kafka and Spark Streaming, and familiarity with containerization tools like Docker or Kubernetes. At Virtusa, teamwork, quality of life, professional and personal development are valued. Joining a global team of 27,000 people, you will have exciting projects, opportunities, and work with state-of-the-art technologies throughout your career. Collaboration and a dynamic team environment are key aspects of Virtusa, providing an environment to nurture new ideas and foster excellence.,

You are a highly skilled and experienced Data Architect with expertise in cloud-based solutions. You will be responsible for designing, implementing, and optimizing data architecture to meet the organization's current and future needs. Your role will involve data modeling, transformation, governance, and hands-on experience with modern cloud platforms and tools such as Snowflake, Spark, Data Lakes, and Data Warehouses. Collaboration with cross-functional teams and stakeholders is crucial, and you will establish and enforce standards and guidelines across data platforms to ensure consistency, scalability, and best practices. You will be accountable for architecting and implementing scalable, secure, and high-performance cloud data platforms that integrate data lakes, data warehouses, and databases. Developing comprehensive data models to support analytics, reporting, and operational needs will be a key responsibility. Leading the design and execution of ETL/ELT pipelines to process and transform data efficiently using tools like Talend, Matillion, SQL, BigData, Hadoop, AWS EMR, and Apache Spark is essential. You will integrate diverse data sources into cohesive and reusable datasets for business intelligence and machine learning purposes. Establishing, documenting, and enforcing standards and guidelines for data architecture, data modeling, transformation, and governance across all data platforms will be part of your role. Ensuring consistency and best practices in data storage, integration, and security throughout the organization is critical. You will establish and enforce data governance standards to ensure data quality, security, and compliance with regulatory requirements, implementing processes and tools to manage metadata, lineage, and data access controls. Your expertise will be utilized in utilizing Snowflake for advanced analytics and data storage needs, optimizing performance and cost efficiency. Leveraging modern cloud platforms to manage data lakes and ensure seamless integration with other services is also a key responsibility. Collaboration with business stakeholders, data engineers, and analysts to gather requirements and translate them into technical designs is essential, along with effectively communicating architectural decisions, trade-offs, and progress to both technical and non-technical audiences. Continuous improvement is part of your role, where you will stay updated on emerging trends in cloud and data technologies, recommending innovations to enhance the organization's data capabilities and optimizing existing architectures to improve scalability, performance, and maintainability. Your technical skills should include expertise in data modeling, data architecture design principles, Talend, Matillion, SQL, BigData, Hadoop, AWS EMR, Apache Spark, Snowflake, and cloud-based data platforms. Experience with data lakes, data warehouses, relational and NoSQL databases, data transformation techniques, ETL/ELT pipelines, DevOps/DataOps/MLOps tools, and standards and governance frameworks is necessary. You should have exceptional written and verbal communication skills to interact effectively with technical teams and business stakeholders. Ideally, you should have 5+ years of experience in data architecture focusing on cloud technologies, a proven track record of delivering scalable, cloud-based data solutions, and a Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. Preferred qualifications include certifications in Snowflake, AWS data services, any RDBMS/NoSQL, AI/ML, Data Governance, familiarity with machine learning workflows and data pipelines, and experience working in Agile development environments.,

Design, build, and maintain data pipelines on the AWS platform. Work with AWS services like S3, Glue, EMR, and Redshift. Process and analyze large datasets to support business insights. Ensure data quality, integrity, and security in the data lake. Location - Pan India.

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing streaming pipelines Experience to work with Hadoop / AWS eco system components to implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Kafka, any Cloud computing etc Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Total 3-5+ years of experience in Data Management (DW, DL, Data Platform, Lakehouse) and Data Engineering skills Minimum 4+ years of experience in Big Data technologies with extensive data engineering experience in Spark / Python or Scala ; Minimum 3 years of experience on Cloud Data Platforms on AWS; Experience in AWS EMR / AWS Glue / DataBricks, AWS RedShift, DynamoDB Good to excellent SQL skills Preferred technical and professional experience Certification in AWS and Data Bricks or Cloudera Spark Certified developers

The role involves working as part of the Infosys delivery team with a focus on quality assurance issue resolution and ensuring high customer satisfaction. Your responsibilities will include interfacing with clients, understanding requirements, creating and reviewing designs, validating architecture, and providing high levels of service offerings in the technology domain. Additionally, you will be involved in project estimation, solution delivery, technical risk planning, code reviews, unit test plan reviews, team leadership, knowledge management, and adherence to organizational guidelines and processes. As a key contributor, you will play a significant role in developing efficient programs and systems to support clients in their digital transformation journey. If you have the skills and expertise in AWS EMR, Big Data, Data Processing, and Map Reduce, and are passionate about delivering optimized high-quality code deliverables, this opportunity is tailored for you. Join us and be a part of helping our clients navigate their digital transformation journey.,

As a skilled professional with over 5 years of experience in programming with Python, you will be responsible for designing, developing, and implementing performant ETL pipelines using Python API (pySpark) of Apache Spark on AWS EMR. Your role will involve hands-on experience in developing ETL data pipelines, configuring EMR clusters on AWS, and working with AWS S3 object storage from Spark. You should have a strong proficiency in Python, familiarity with functional programming concepts, and data modeling. Your key responsibilities will include understanding Spark's Data frame and API, troubleshooting Spark jobs, and monitoring spark jobs using Spark UI. The ideal candidate should have at least 3 years of hands-on experience in developing ETL data pipelines using pySpark on AWS EMR and a good understanding of Spark's Data frame and API. This is a full-time, permanent position suitable for someone with a solid background in ETL, Python programming, Apache Spark, pySpark, AWS S3, and Data Modeling. The work location is in person, and the application deadline is 22/02/2025. Benefits include the flexibility to work from home, and the schedule consists of day shift, fixed shift, and morning shift. If you have a strong background in ETL, Python, Apache Spark, pySpark, AWS S3, and Data Modeling, we encourage you to apply for this exciting opportunity.,