281 Spark Streaming Jobs - Page 3

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Must have minimum of 12 15 years of experience implementing batch and streaming applications using Spark and Scala Experience with Confluent or Apache Kafka and Connectors Experience with Java and building microservices, web applications or other applications Develop data pipelines responsible for ingesting large amounts of data from various sources Help evolve data architecture and work on Next Generation real time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems Write code and develop worker node for business logic, ETL and orchestration processes Develop algorithms for better attribution rules and category classifiers Work...

Confluent CDC System Analyst Role Overview: We are seeking an experienced Confluent Change Data Capture (CDC) System Analyst/ Tech lead to implement real-time data streaming solutions. The role involves implementing robust CDC frameworks using Confluent Kafka, ensuring seamless data integration between core banking systems and analytics platforms. The ideal candidate will have deep expertise in event-driven architectures, CDC technologies, and cloud-based data solutions. Key Responsibilities: Implement Confluent Kafka-based CDC solutions to support real-time data movement across banking systems. Implement event-driven and microservices-based data solutions for enhanced scalability, resilienc...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Overview This position is for Manager Data Engineering in the Commercial Data as a Service group. In this position you will enjoy being responsible for helping define and maintain the data systems key to delivering successful outcomes for our customers. will lead a software development team focused on our core services someone who combines deep technical experience with great people management skills and can help elevate our teams and our approach to engineering. We are building a culture that fosters innovation, built on a foundation of inclusion and diversity. We are looking for a candidate that has proven experience in coding, architecture leadership, and leading engineers in a fast-paced...

EPAM has presence across 40+ countries globally with 55,000 + professionals & numerous delivery centers, Key locations are North America, Eastern Europe, Central Europe, Western Europe, APAC, Mid East & Development Centers in India (Hyderabad, Pune & Bangalore) . Location: Chennai/Coimbatore Work Mode: Hybrid (2-3 days office in a week) Full time Job Description: 5-14 Years of in Big Data & Data related technology experience Expert level understanding of distributed computing principles Expert level knowledge and experience in Apache Spark Hands on programming with Python/Scala/Java Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming...

Job Opening with a Big 4 management consulting. Position: Senior Developer (Kafka / Spark Streaming) Experience: 6-8 Years Required Qualifications & Experience- Minimum experience of working in projects as a Senior Azure Data Engineer. B.Tech/B.E degree in Computer Science or Information Technology. Extensive working experience in streaming services like Kafka/Spark Streaming/Azure Event Hub/ Azure IOT Hub. Analyzing real time data pipelines using python, Spark Streaming and data base tools like SQL, Mongo DB for processing data from different sources. Strong understanding of Apache Hadoop-based analytics Working knowledge of Docker and Kubernetes

Position: Senior Developer (Kafka / Spark Streaming) Experience: 6-8 Years No. Of Position: 1 Locations- Mumbai, Bangalore Required Qualifications & Experience- Minimum experience of working in projects as a Senior Azure Data Engineer. B.Tech/B.E degree in Computer Science or Information Technology. Extensive working experience in streaming services like Kafka/Spark Streaming/Azure Event Hub/ Azure IOT Hub. Analyzing real time data pipelines using python, Spark Streaming and data base tools like SQL, Mongo DB for processing data from different sources. Strong understanding of Apache Hadoop-based analytics Working knowledge of Docker and Kubernetes Experience with Platform Automation tools (D...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

As a Senior Developer specializing in Databricks, your role will involve working offshore and collaborating with a team to design solutions, perform code reviews, and mentor junior engineers. Your experience in SQL, backend development, and data-driven projects will be crucial for success in this position. You will be responsible for utilizing your expertise in Python/PySpark, SQL, Databricks, SCALA, Spark/Spark Streaming, Big Data Tool Set, Linux, and Kafka to contribute to the development projects. Key Responsibilities: - Designing solutions, performing code reviews, and mentoring junior engineers - Working on data-driven projects with a focus on strong SQL and backend experience - Collabo...

As a Data Engineer at our company, you will have the following responsibilities: - Review and understand business requirements to ensure development tasks are completed within the provided timeline and are thoroughly tested with minimal defects. - Collaborate with a software development team to implement best practices and optimize the performance of Data applications. - Engage with various teams across the company and interact with customers to define, design, and showcase new concepts and solutions. - Conduct research on new Big Data technologies, evaluating their maturity and alignment with business and technology strategies. - Work closely with other developers to meet client needs effic...

Experience: 4 to 10 Years Location: Mumbai / Bangalore (Hybrid / Onsite) Employment Type: Full-Time About the Role We are looking for skilled backend engineers proficient in Scala , Java , or PySpark , to join a dynamic global team working for a leading European & USA BFSI client . This role is ideal for professionals who enjoy solving complex data and performance challenges, building high-volume, low-latency financial systems , and working with modern data processing frameworks such as Apache Spark and Scala . Youll be part of a high-impact engineering team , collaborating with architects, data engineers, and product stakeholders across regions. Key Responsibilities Design, develop, and opt...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are currently considering only candidates based in Bangalore who can attend in-person interviews after CV shortlisting. We're open to considering candidates with 3 to 12 years of experience, provided you are a hands-on coder with strong technical skills. The Modern Data Engineer is responsible for designing, implementing, and maintaining scalable data architectures using cloud technologies, primarily on AWS, to support the next evolutionary stage of the Investment Process. They build robust data pipelines, optimize data storage, and access patterns, and ensure data quality while collaborating across engineering teams to deliver high-value data products Key Responsibilities • Implement and...

As a Technical Lead in Data Engineering for TurboRater RQR: CPQ Rating at Zywave in Pune, India, you will be responsible for leading the design and implementation of scalable, secure, and high-performance data pipelines that power rating and CPQ capabilities. Your deep technical expertise combined with insurance domain knowledge will be crucial in driving innovation and delivering business impact. **Key Responsibilities:** - Lead end-to-end design and development of data pipelines supporting TurboRater RQR: CPQ Rating. - Architect and implement Medallion Architecture (Bronze, Silver, Gold layers) for structured and semi-structured data. - Drive adoption of Data Mesh principles, decentralizin...

Data Engineer to build and manage data pipelines that support batch and streaming data solutions. The role requires expertise in creating seamless data flows across platforms like Data Lake/Lakehouse in Cloudera, Azure Databricks, Kafka for both batch and stream data pipelines etc. Responsibilities: Strong experience in develop, test, and maintain data pipelines (batch & stream) using Cloudera, Spark, Kafka and Azure services like ADF, Cosmos DB, Databricks, NoSQL DB/ Mongo DB etc. Strong programming skills in spark, python or scala & SQL. Optimize data pipelines to improve speed, performance, and reliability, ensuring that data is available for data consumers as required. Create ETL pipelin...

Role & responsibilities Design and build a scalable data platform that can ingest, store, manage and stream massive amounts of data while simplifying its analysis and processing to enable rapid development of high-quality data products and services. •Implement and test robust low latency data pipelines for preparing/curating data that will support various data as a service product. •You will work with various cross functional teams to explore data and figure out all the data wrangling needed to clean, curate the data and build the end-to-end data pipeline per requirements. •You will design necessary privacy and security preventive and detective controls into the CICD pipelines of each data p...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Job Description Primary Skill: Spark, Java programming Strong knowledge in Apache Spark framework Core Spark, Spark Data Frames, Spark streaming Hands-on experience in any one of the programming languages (Java, Scala) Good understanding of distributed programming concepts. Experience in optimizingSparkDAG, and Hive queries on Tez Experience using tools like Git, Autosys, Bitbucket, Jira Mandatory Skills: Apache Spark . Experience: 5-8 Years .

Pyspark Lead Position Overview We are seeking a skilled and experienced Senior PySpark Developer with expertise in Apache spark,Spark Batch, and Spark Streaming to join our dynamic team. The ideal candidate will design, develop, and maintain high-performance, scalable applications for processing large-scale data in batch and real-time environments. Required Skills and Qualifications Experience :7+ years of professional experience in PySpark development. Technical Skills : Strong proficiency in PySpark Deep understanding of Apache Spark architecture, distributed computing concepts and parallel processing. Proven experience in building and optimizing complex ETL pipeline using PySpark Experien...

SUMMARY We are seeking a highly experienced and passionate Data Engineer to join our growing data team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining our robust data infrastructure, ensuring data quality, availability, and scalability. You will play a critical role in empowering our data-driven decision-making by building and optimizing data pipelines, ETL processes, and data warehousing solutions. This is a senior-level position requiring extensive experience with database technologies, data modelling, and cloud platforms. Responsibilities: Data Pipeline Development & Maintenance: Design, build, and maintain end-to-end data pipelines for inges...

Skills- AWS Data Engineer Experience: 9 to 13 years Location: AIA-Pune We are seeking skilled and dynamic Cloud Data Engineers specializing in AWS, Databricks. The ideal candidate will have a strong background in data engineering, with a focus on data ingestion, transformation, and warehousing. They should also possess excellent knowledge of PySpark or Spark, and a proven ability to optimize performance in Spark job executions. Key Responsibilities: - Design, build, and maintain scalable data pipelines for a variety of cloud platforms including AWS. - Implement data ingestion and transformation processes to facilitate efficient data warehousing. - Utilize cloud services to enhance data proce...