34 Spark Core Jobs

Job Description Primary Skill: Spark, Java programming Strong knowledge in Apache Spark framework Core Spark, Spark Data Frames, Spark streaming Hands-on experience in any one of the programming languages (Java, Scala) Good understanding of distributed programming concepts. Experience in optimizingSparkDAG, and Hive queries on Tez Experience using tools like Git, Autosys, Bitbucket, Jira Mandatory Skills: Apache Spark . Experience: 5-8 Years .

About the Role: We are looking for a skilled and passionate Spark Developer to design, build, and optimize large-scale data processing systems. The ideal candidate will have strong expertise in Apache Spark , Big Data ecosystems , and distributed computing frameworks . Key Responsibilities: Design, develop, and maintain data processing pipelines using Apache Spark (Core, SQL, Streaming, Structured Streaming) . Implement ETL workflows for data ingestion, transformation, and integration across multiple data sources. Optimize Spark applications for performance, scalability, and reliability in distributed environments. Work with large structured and unstructured datasets to support analytical an...

Skills-Java, Spark, SQL Experience: 6 to 12 years Location: AIA-Hyderabad Responsibilities and JD in brief along with additional criteria to be considered (if any): At least 6+ years of experience and strong knowledge in Java and related frameworks. Expertise in building robust, scalable and maintainable applications with Java and Spark . Able to write clean, maintainable and efficient Java code following best practices. Having deep understanding of Java core and Java EE concepts, JVM internals such as class loading and memory management. At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies....

Looking for a skilled professional with expertise in Spark and Delta Lake to join our team. The ideal candidate will have 7-12 years of experience. Roles and Responsibility Design and implement data pipelines using Spark and Delta Lake as the data storage layer. Develop and optimize Spark applications using Python, Scala, or Java. Work with large datasets to ensure data quality and integrity. Implement data partitioning and schema evolution techniques using Delta Lake. Collaborate with cross-functional teams to integrate Spark applications with ETL processes. Troubleshoot and resolve issues related to Spark application performance and data quality. Job Requirements Strong understanding of Sp...

As an experienced candidate with over 8 years of overall experience and 5+ years of relevant experience, your role will involve the following key responsibilities: - Strong understanding of Spark core including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Proficiency in Delta Lake features such as time travel, schema evolution, and data partitioning - Designing and building data pipelines using Spark and Delta Lake - Solid experience in Python/Scala/Java for Spark development - Knowledge of data ingestion from files, APIs, and databases - Familiarity with data validation and quality best practices - Working knowledge of data warehouse concepts and data modeling - Hands-on expe...

Role Overview: As a seasoned Spark Developer with 5+ years of experience, you have a deep understanding of developing, testing, deploying, and debugging Spark Jobs using Scala on the Hadoop Platform. Your expertise includes proficiency in Spark Core and working with RDDs and Spark SQL, knowledge of Spark Optimization Techniques and Best practices, familiarity with Scala Functional Programming concepts like Try, Option, Future, and Collections, understanding of Scala OOPS principles including Classes, Traits, Objects (Singleton and Companion), and Case Classes, strong grasp of Scala Language Features such as Type System and Implicit/Givens, hands-on experience in Hadoop Environment (HDFS/Hive...

Role Overview: You will be joining the ICG TTS Operations Technology (OpsTech) Group to assist in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role as a Lead Spark Scala Engineer will involve managing development teams in the distributed systems Eco-System and collaborating effectively as a team player. Your primary objective will be to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical leadership and guidance to a team of ...

As an Azure DB Architect with 8+ years of experience, your role will involve: - Designing Databricks workspaces and Unity Catalog for multi-tenant environments - Architecting Lakehouse solutions with Delta Lake - Demonstrating expertise in Spark core for real-time and batch pipeline designing - Implementing data modelling, security & governance using RBAC and ABAC - Ensuring data lineage and metadata management - Orchestrating workflows using Databricks Workflows - Applying best practices for artifact organization and naming conventions in Databricks Qualifications required for this role include: - Minimum of 10 years of experience in the field - Deep understanding of AWS services such as Re...

Looking for a skilled professional with expertise in Spark and Delta Lake to join our team. The ideal candidate will have 7-12 years of experience. Roles and Responsibility Design and implement data pipelines using Spark and Delta Lake as the data storage layer. Develop and optimize Spark applications using Python, Scala, or Java. Work with large datasets to ensure data quality and integrity. Implement data partitioning and schema evolution techniques using Delta Lake. Collaborate with cross-functional teams to integrate Spark applications with ETL processes. Troubleshoot and resolve issues related to Spark application performance and data quality. Job Requirements Strong understanding of Sp...

Role Overview: You will be working as a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group to help in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role will involve managing development teams in the distributed systems Eco-System and being a strong team player. It is expected that you will have superior technical knowledge of current programming languages, technologies, and other leading-edge development tools to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal perf...

Primary Skill: Spark, Java programming Strong knowledge in Apache Spark framework Core Spark, Spark Data Frames, Spark streaming Hands-on experience in any one of the programming languages (Java, Scala) Good understanding of distributed programming concepts. Experience in optimizing Spark DAG, and Hive queries on Tez Experience using tools like Git, Autosys, Bitbucket, Jira Mandatory Skills: Apache Spark.Experience: 5-8 Years.

Role & responsibilities We are looking for a candidate with strong experience into Spark Core/Streaming, Spark RDD API, Spark Dataframes API with Scala with Machine Learning and AI, Should have hands-on programming experience and Good Understanding of Data Structures, Algorithms, Data Transformation, Data Ingestion, Optimization mechanism/techniques, Good understanding of Big Data (Hadoop, MapReduce, Kafka, Cassandra) Technologies. We deal with huge amounts of data at a massive scale, so we are looking for engineers who love solving challenging problems through conducting independent research and collaborating with teams across our product teams to help improve the overall product experience...

Responsibilities * Design, develop, test, and deploy Spark Streaming applications using Scala/Spark Core. * Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions. * Troubleshoot issues in Spark Streaming applications and optimize performance for improved efficiency. * Stay up-to-date with the latest developments in Spark Streaming and contribute to its evolution. Job Requirements * 3-10 years of experience in developing Spark Streaming applications using Scala/Spark Core. * Strong understanding of Spark programming concepts, including RDDs, Dataframes, Datasets, etc. * Experience with big data processing technologies such as Hadoop/Hive/Pig is a pl...

As a Senior Python Developer specializing in Python and Spark programming, you will be responsible for leveraging your thorough and hands-on knowledge to write efficient, reusable, and reliable Python code. You should possess a minimum of 5+ years of experience in Python programming and have a strong proficiency in both Python and Spark programming. Your key responsibilities will include implementing Spark Core, Spark SQL, and Spark Streaming, working with Spark within the Hadoop Ecosystem, and designing and implementing low-latency, high-availability, and performance applications. Additionally, you will lead and guide a team of junior Python developers, collaborate with other team members a...

This position falls under the ICG TTS Operations Technology (OpsTech) Group, focusing on assisting in the implementation of a next-generation Digital Automation Platform and Imaging Workflow Technologies. The ideal Candidate should have relevant experience in managing development teams within the distributed systems Eco-System and must exhibit strong teamwork skills. The candidate is expected to possess superior technical knowledge of current programming languages, technologies, and leading-edge development tools. The primary objective of this role is to contribute to applications, systems analysis, and programming activities. As a Lead Spark Scala Engineer, the candidate should have hands-o...

Role & responsibilities We are looking for a candidate with strong experience into Spark Core/Streaming, Spark RDD API, Spark Dataframes API with Scala with Machine Learning and AI, Should have hands-on programming experience and Good Understanding of Data Structures, Algorithms, Data Transformation, Data Ingestion, Optimization mechanism/techniques, Good understanding of Big Data (Hadoop, MapReduce, Kafka, Cassandra) Technologies. We deal with huge amounts of data at a massive scale, so we are looking for engineers who love solving challenging problems through conducting independent research and collaborating with teams across our product teams to help improve the overall product experience...

EPAM has presence across 40+ countries globally with 55,000 + professionals & numerous delivery centers, Key locations are North America, Eastern Europe, Central Europe, Western Europe, APAC, Mid East & Development Centers in India (Hyderabad, Pune & Bangalore). Location: Gurgaon/Pune/Hyderabad/Bengaluru/ChennaiWork Mode: Hybrid (2-3 days office in a week) Job Description: 5-14 Years of in Big Data & Data related technology experience Expert level understanding of distributed computing principles Expert level knowledge and experience in Apache Spark Hands on programming with Python Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming ...

We are seeking an experienced Software Engineer with deep expertise in Scala programming and Big Data technologies to design, develop, and maintain large-scale distributed data processing systems. The ideal candidate will be a hands-on developer with a strong understanding of data pipelines, Spark ecosystem, and related technologies, capable of delivering clean, efficient, and scalable code in an Agile environment. Key Responsibilities Develop and maintain scalable, efficient, and robust data processing pipelines using Scala and Apache Spark (Spark Core, Spark SQL, Spark Streaming). Write clean, maintainable, and well-documented Scala code following industry best practices and coding standar...

The Team The Data Engineering team is responsible for architecting, building, and maintaining our evolving data infrastructure, as well as curating and governing the data assets created on our platform. We work closely with various stakeholders to acquire, process, and refine vast datasets, focusing on creating scalable and optimized data pipelines. Our team possesses broad expertise in critical data domains, technology stacks, and architectural patterns. We foster knowledge sharing and collaboration, resulting in a unified strategy and seamless data management. The Impact: This role is the foundation of the products delivered. The data onboarded is the base for the company as it feeds into ...

As a software developer, you will be working in a constantly evolving environment driven by technological advances and the strategic direction of the organization you are employed by. Your primary responsibilities will include creating, maintaining, auditing, and enhancing systems to meet specific needs, often based on recommendations from systems analysts or architects. You will be tasked with testing both hardware and software systems to identify and resolve system faults. Additionally, you will be involved in writing diagnostic programs and designing and developing code for operating systems and software to ensure optimal efficiency. In situations where necessary, you will also provide re...

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, a...

At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career, Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express, Join Team Amex and let's lead the way together, From building next-generation apps and microservices in Kotlin to using AI to help protect our franchise and custome...

Experience: 4-10years Location - Pune, Bangalore, Chennai, Noida and Gurgaon Notice period should be Immediate to 30days only Mandatory skill - Apache Spark, JAVA programming Strong knowledge in Apache Spark framework Core Spark, Spark Data Frames, Spark streaming Hands-on experience in any one of the programming languages (Java) Good understanding of distributed programming concepts Experience in optimizing Spark DAG, and Hive queries on Tez Experience using tools like Git, Autosys, Bitbucket, Jira Ability to apply DWH principles within Hadoop environment and NoSQL databases.

Responsibilities: * Design, develop, test & maintain Scala applications using Spark. * Collaborate with cross-functional teams on project delivery. * Optimize application performance through data analysis.

Mandatory Skill : Spark and Scala Data Engineering. Secondary Skill Python 5+ years of in depth hands on experience of developing, testing, deployment and debugging of Spark Jobs using Scala in Hadoop Platform In depth knowledge of Spark Core, working with RDDs, Spark SQL In depth knowledge on Spark Optimization Techniques and Best practices Good Knowledge of Scala Functional Programming: Try, Option, Future, Collections Good Knowledge of Scala OOPS: Classes, Traits and Objects (Singleton and Companion), Case Classes Good Understanding of Scala Language Features: Type System, Implicit/Givens Hands on experience of working in Hadoop Environment (HDFS/Hive), AWS S3, EMR Working experience on W...