268 Aws Emr Jobs - Page 6

As a Python Developer joining our team in partnership with a renowned financial institution, you will be responsible for analyzing existing SAS DI pipelines and SQL-based transformations. Your role will involve translating and optimizing SAS SQL logic into Python code using frameworks like Pyspark. You will play a crucial part in developing and maintaining scalable ETL pipelines using Python on AWS EMR, implementing data transformation, cleansing, and aggregation logic to meet business requirements. Your expertise will be utilized to design modular and reusable code for distributed data processing tasks on EMR clusters. You will also be integrating EMR jobs with upstream and downstream syste...

We are searching for a highly skilled and seasoned Senior ETL & Data Streaming Engineer with over 10 years of experience to take on a crucial role in the design, development, and maintenance of our robust data pipelines. The ideal candidate will possess in-depth expertise in batch ETL processes as well as real-time data streaming technologies, along with extensive hands-on experience with AWS data services. A proven track record of working with Data Lake architectures and traditional Data Warehousing environments is a must. Your responsibilities will include designing, developing, and implementing highly scalable, fault-tolerant, and performant ETL processes using leading ETL tools to extrac...

You are a skilled and proactive Python / PySpark Developer sought to join the data engineering or analytics team. Your responsibilities will include building scalable data pipelines, performing large-scale data processing, and collaborating with data scientists, analysts, and business stakeholders. You will design, develop, and optimize ETL data pipelines using PySpark on big data platforms (e.g., Hadoop, Databricks, EMR). Writing clean, efficient, and modular code in Python for data processing and integration tasks is essential. Working with large datasets to extract insights, transform raw data, and ensure data quality will be part of your daily tasks. Collaborating with cross-functional t...

You are a highly skilled and experienced Data Architect with expertise in cloud-based solutions. You will be responsible for designing, implementing, and optimizing data architecture to meet the organization's current and future needs. Your role will involve data modeling, transformation, governance, and hands-on experience with modern cloud platforms and tools such as Snowflake, Spark, Data Lakes, and Data Warehouses. Collaboration with cross-functional teams and stakeholders is crucial, and you will establish and enforce standards and guidelines across data platforms to ensure consistency, scalability, and best practices. You will be accountable for architecting and implementing scalable, ...

Design, build, and maintain data pipelines on the AWS platform. Work with AWS services like S3, Glue, EMR, and Redshift. Process and analyze large datasets to support business insights. Ensure data quality, integrity, and security in the data lake. Location - Pan India.

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

The role involves working as part of the Infosys delivery team with a focus on quality assurance issue resolution and ensuring high customer satisfaction. Your responsibilities will include interfacing with clients, understanding requirements, creating and reviewing designs, validating architecture, and providing high levels of service offerings in the technology domain. Additionally, you will be involved in project estimation, solution delivery, technical risk planning, code reviews, unit test plan reviews, team leadership, knowledge management, and adherence to organizational guidelines and processes. As a key contributor, you will play a significant role in developing efficient programs a...

As a skilled professional with over 5 years of experience in programming with Python, you will be responsible for designing, developing, and implementing performant ETL pipelines using Python API (pySpark) of Apache Spark on AWS EMR. Your role will involve hands-on experience in developing ETL data pipelines, configuring EMR clusters on AWS, and working with AWS S3 object storage from Spark. You should have a strong proficiency in Python, familiarity with functional programming concepts, and data modeling. Your key responsibilities will include understanding Spark's Data frame and API, troubleshooting Spark jobs, and monitoring spark jobs using Spark UI. The ideal candidate should have at le...

Role: Lead Data Engineer Location: kochi/Trivandrum Experience: 7+ years Skill Set: Python, pyspark, AWS Budget: Max 25lpa Job Overview We are seeking an experienced Senior Data Engineer to lead the development of a scalable data ingestion framework while ensuring high data quality and validation. The successful candidate will also be responsible for designing and implementing robust APIs for seamless data integration. This role is ideal for someone with deep expertise in building and managing big data pipelines using modern AWS-based technologies, and who is passionate about driving quality and efficiency in data processing systems. Key Responsibilities Data Ingestion Framework: o Design & ...

As a Senior Cloud Data Developer, you will play a crucial role in bridging the gap between data engineering and cloud-native application development. Your primary responsibility will be to design and maintain scalable data solutions in the cloud by leveraging cutting-edge technologies and modern development practices. You will be tasked with developing data-centric applications using Java Spring Boot and various AWS services. Your expertise will be essential in creating and managing efficient data pipelines, ETL processes, data workflows, and orchestrating data processing systems. Real-time data processing solutions using AWS SNS/SQS and AWS Pipes will be a key part of your role. Additionall...

Join us as a Data Engineer. You'll be the voice of our customers, using data to tell their stories and put them at the heart of all decision-making. We will look to you to drive the build of effortless, digital-first customer experiences. If you're ready for a new challenge and want to make a far-reaching impact through your work, this could be the opportunity you're looking for. This role is based in India, and as such, all normal working days must be carried out in India. We're offering this role at vice president level. As a Data Engineer, you'll be looking to simplify our organization by developing innovative data-driven solutions through data pipelines, modeling, and ETL design, aiming ...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too Required education Bachelo...

About the Role: OSTTRA India The Role Enterprise Architect - Integration The Team The OSTTRA Technology teamis composed of Capital Markets Technology professionals, who build,supportand protect the applications that operate our network. The technology landscapeincludeshigh-performance, high-volume applications as well as compute intensive applications,leveragingcontemporary microservices, cloud-based architectures. The Impact: Together, we build, support, protect and manage high-performance, resilient platforms that process more than 100 million messages a day. Our services are vital to automated trade processing around the globe, managing peak volumes and working with our customers and regu...

Oracle Cloud is a comprehensive enterprise-gradeplatform that offers best-in-class services across Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS). Oracle Cloud platform offers choice and flexibility for customers to build, deploy, integrate, and extend applications in the cloud that enable adapting to rapidly changing business requirements, promote interoperability and avoid lock-in. This platform supports numerous open standards (SQL, HTML5, REST, and more), open-source solutions (such as Kubernetes, Hadoop, Spark and Kafka) and a wide variety of programming languages, databases, tools and integration frameworks. You're Opportunity: Values ...

We are seeking a dynamic and experienced Tech Lead with a strong foundation in Java and Apache Spark to join our team. In this role, you will lead the development and deployment of scalable cloud-based data solutions, leveraging your expertise in AWS and big data technologies. Key Responsibilities: Lead the design, development, and deployment of scalable and reliable data processing solutions on AWS using Java and Spark. Architect and implement big data processing pipelines using Apache Spark on AWS EMR. Develop and deploy Serverless applications using AWS Lambda, integrating with other AWS services. Utilize Amazon EKS for container orchestration and microservices management. Design and impl...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

At Capgemini Invent, we believe difference drives change. As inventive transformation consultants, we blend our strategic, creative and scientific capabilities, collaborating closely with clients to deliver cutting-edge solutions. Join us to drive transformation tailored to our client's challenges of today and tomorrow. Informed and validated by science and data. Superpowered by creativity and design. All underpinned by technology created with purpose. Your Role IT experience with a minimum of 5+ years of experience in creating data warehouses, data lakes, ETL/ELT, data pipelines on cloud. Has data pipeline implementation experience with any of these cloud providers - AWS, Azure, GCP. prefer...

Capgemini Invent Capgemini Invent is the digital innovation, consulting and transformation brand of the Capgemini Group, a global business line that combines market leading expertise in strategy, technology, data science and creative design, to help CxOs envision and build what's next for their businesses. Your Role Should have developed/Worked for atleast 1 Gen AI project. Has data pipeline implementation experience with any of these cloud providers - AWS, Azure, GCP. Experience with cloud storage, cloud database, cloud data warehousing and Data lake solutions like Snowflake, Big query, AWS Redshift, ADLS, S3. Has good knowledge of cloud compute services and load balancing. Has good knowled...

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs.Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too Required education Bachelor...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

ROLE PROFILE: Are you ready to take your engineering career to the next level Join us as a Mid Lead Engineer and contribute to building state-of-the-art data platforms in AWS , leveraging Python and Spark . Be part of a dynamic team, driving innovation and scalability for data solutions in a supportive and hybrid work environment. ROLE SUMMARY: This role is ideal for an experienced data engineer looking to step into a leadership position while remaining hands-on with cutting-edge technologies. You will design, implement and optimize ETL workflows using Python and Spark, contributing to our robust data lakehouse architecture on AWS. Success in this role requires technical expertise, strong pr...