208 Spark Streaming Jobs

SUMMARY We are seeking a highly experienced and passionate Data Engineer to join our growing data team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining our robust data infrastructure, ensuring data quality, availability, and scalability. You will play a critical role in empowering our data-driven decision-making by building and optimizing data pipelines, ETL processes, and data warehousing solutions. This is a senior-level position requiring extensive experience with database technologies, data modelling, and cloud platforms. Responsibilities: Data Pipeline Development & Maintenance: Design, build, and maintain end-to-end data pipelines for inges...

Skills- AWS Data Engineer Experience: 9 to 13 years Location: AIA-Pune We are seeking skilled and dynamic Cloud Data Engineers specializing in AWS, Databricks. The ideal candidate will have a strong background in data engineering, with a focus on data ingestion, transformation, and warehousing. They should also possess excellent knowledge of PySpark or Spark, and a proven ability to optimize performance in Spark job executions. Key Responsibilities: - Design, build, and maintain scalable data pipelines for a variety of cloud platforms including AWS. - Implement data ingestion and transformation processes to facilitate efficient data warehousing. - Utilize cloud services to enhance data proce...

Key Responsibilities : Design, implement, and maintain data science pipelines from data ingestion to model deployment. Collaborate with data scientists to operationalize ML models and algorithms in production environments. Develop robust APIs and services for ML model inference and integration. Build and optimize large-scale data processing systems using Spark, Pandas, or similar tools. Ensure data quality and pipeline reliability through rigorous testing, validation, and monitoring. Work with cloud infrastructure (AWS) for scalable ML deployment. Manage model versioning, feature engineering workflows, and experiment tracking. Optimize performance of models and pipelines for latency, cost, a...

About the Role: We are looking for a skilled and passionate Spark Developer to design, build, and optimize large-scale data processing systems. The ideal candidate will have strong expertise in Apache Spark , Big Data ecosystems , and distributed computing frameworks . Key Responsibilities: Design, develop, and maintain data processing pipelines using Apache Spark (Core, SQL, Streaming, Structured Streaming) . Implement ETL workflows for data ingestion, transformation, and integration across multiple data sources. Optimize Spark applications for performance, scalability, and reliability in distributed environments. Work with large structured and unstructured datasets to support analytical an...

Role Overview: As a Solution Designer (Cloud Data Integration) at Barclays, you will be responsible for supporting the successful delivery of location strategy projects to plan, budget, agreed quality, and governance standards. You will spearhead the evolution of the digital landscape, driving innovation and excellence. Your role will involve harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. Key Responsibilities: - Hands-on experience working with large-scale data platforms and developing cloud solutions in the AWS data platform with a proven track record in driving business success. - Strong understanding of AWS and distribute...

You have a strong background in data engineering or a related field with over 8 years of experience. Your expertise includes hands-on experience with Azure Databricks, Spark, Python/Scala, CICD, and scripting for data processing. You have worked with multiple file formats such as Parquet, Delta, and Iceberg, and have knowledge of streaming technologies like Kafka for real-time data ingestion. Additionally, you are experienced in data governance and data security in Azure, and have a deep understanding of Azure Data Services. Key Responsibilities: - Utilize Azure Databricks, Spark, Python, and Scala for scalable and distributed data processing. - Work with various file formats like Parquet, D...

Skills-Java, Spark, SQL Experience: 6 to 12 years Location: AIA-Hyderabad Responsibilities and JD in brief along with additional criteria to be considered (if any): At least 6+ years of experience and strong knowledge in Java and related frameworks. Expertise in building robust, scalable and maintainable applications with Java and Spark . Able to write clean, maintainable and efficient Java code following best practices. Having deep understanding of Java core and Java EE concepts, JVM internals such as class loading and memory management. At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies....

As a skilled and proactive Python / PySpark Developer at our company, you will join our data engineering or analytics team. Your primary responsibility will be to build scalable data pipelines, perform large-scale data processing, and collaborate with data scientists, analysts, and business stakeholders. Key Responsibilities: - Design, develop, and optimize ETL data pipelines using PySpark on big data platforms (e.g., Hadoop, Databricks, EMR). - Write clean, efficient, and modular code in Python for data processing and integration tasks. - Work with large datasets to extract insights, transform raw data, and ensure data quality. - Collaborate with cross-functional teams to understand busines...

As an experienced professional with over 7+ years of experience in Noida, your role will involve the following key responsibilities: - Reviewing and understanding business requirements to ensure timely completion of development tasks with minimal defects - Collaborating with a software development team to implement best practices and enhance the performance of Data applications for meeting client needs - Engaging with various teams and customers to translate their business challenges into innovative solutions - Conducting research on new Big Data technologies to align them with the business and technology strategy - Operating within a rapid and agile development process to accelerate speed t...

Role & responsibilities Overall, 6 to 8 years of experience in IT Industry. Min 6 years of experience working on Data Engineering. Design and develop solution in combination of Cloudera, Palantir and Databricks. Develop Python-based automation scripts to optimize workflows and processes. Integrate Databricks with AWS ecosystem components such as S3, Glue, Athena, Redshift, and Lambda Define architecture for data-driven solutions. Provide solutioning for web applications and data pipelines. Solutioning data analytics requirements with Palantir. Stay updated with emerging technologies and quickly adapt to new tools and frameworks. Work with CI/CD pipelines. Ensure scalability, security, and pe...

Spark & Delta Lake Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL and Spark Streaming. Experience with Spark optimization techniques. Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database) Understanding of data quality best practices and data validation techniques. Other Skills: Understanding of data warehouse concepts, data modelling techniques. ...

Big Data Engineer 3+Yrs Immediate Joinees Remote Must Have Skills: Any Cloud, OOP, ETL/ELT, Hive, Scala/Java/React, CI/CD, Kafka/Kinesis, Spark, Spark Streaming, SQL Share cv to Yogitha@ontimesolutions.in or 7406353337

Role Overview: SuperProcure seeks a strategic, hands-on Director - Product Engineering & Technology to lead and scale our engineering team, ensuring timely, high-quality SaaS product delivery aligned with business goals. Ideal for a dynamic leader passionate about solving complex problems in a high-growth environment. Key responsibilities include: Engineering Leadership Define and enforce best practices in architecture, code quality, testing, deployment, and documentation. Drive a culture of ownership, velocity, and continuous improvement. Collaborate with CXO and cross-functional teams to align on customer priorities and product deliverables. Provide thought leadership and drive technology ...

Serko is a cutting-edge tech platform in global business travel & expense technology. When you join Serko, you become part of a team of passionate travellers and technologists bringing people together, using the world's leading business travel marketplace. We are proud to be an equal opportunity employer, we embrace the richness of diversity, showing up authentically to create a positive impact. There's an exciting road ahead of us, where travel needs real, impactful change. With offices in New Zealand, Australia, North America, and China, we are thrilled to be expanding our global footprint, landing our new hub in Bengaluru, India. With a rapid growth plan in place for India, we're hiring p...

As a Data Engineer Offshore at PwC, you will be part of the managed services team focusing on outsourced solutions and supporting clients across various functions. Your role will involve streamlining operations, reducing costs, and improving efficiency for organizations by managing key processes and functions on their behalf. Utilizing your skills in project management, technology, and process optimization, you will deliver high-quality services to clients. Your main responsibilities will include transitioning and running services, managing delivery teams, programs, commercials, performance, and delivery risk. Continuous improvement and optimization of managed services processes, tools, and ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Skill Sets: Must have: Minimum 8 to 10 years of experience in Java, REST API, Spring boot Hands-on experience in Angular JS or React JS or View JS Bachelors degree or higher in computer science, data science, or a related field. Hands-on experience with data cleaning, visualization, and reporting. Experience working in an agile environment. Excellent analytical and problem-solving skills. Excellent communication and interpersonal skills. Good to have: Familiarity with the Hadoop ecosystem Experience with AWS ( EMR) is a plus. At least 2 years of relevant experience with real-time data stream platforms such as Kafka and Spark Streaming.

As a candidate for this role, you would be responsible for the following: - Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. - Experience with Spark optimization techniques. - Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. - Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. - Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. - Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database). - Understanding of data quality best practices and data validation techniques. Qualificatio...

As a Data Engineer 2 at JLL, you will be part of the JLL Technologies Enterprise Data team, working in a diverse and fast-paced environment to develop and implement data solutions that are strategic for the business. Your primary focus will be on developing solutions leveraging cloud big data technology to ingest, process, and analyze large, disparate data sets to meet business requirements. Additionally, you will collaborate with internal and external teams to drive data development and derive insights from a variety of data sources. - Develop solutions using cloud big data technology to process and analyze large data sets - Design and develop data pipelines from various internal and extern...

As an experienced candidate with over 8 years of overall experience and 5+ years of relevant experience, your role will involve the following key responsibilities: - Strong understanding of Spark core including RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Proficiency in Delta Lake features such as time travel, schema evolution, and data partitioning - Designing and building data pipelines using Spark and Delta Lake - Solid experience in Python/Scala/Java for Spark development - Knowledge of data ingestion from files, APIs, and databases - Familiarity with data validation and quality best practices - Working knowledge of data warehouse concepts and data modeling - Hands-on expe...

About the Role Were seeking a highly skilled Data Engineer with a strong development background and a passion for transforming data into valuable insights. The ideal candidate will play a key role in designing, building, and maintaining scalable data pipelines and analytics solutions that support critical business decisions. What youll be doing Design, build, and optimize robust data pipelines for large-scale data processing Write complex SQL queries for data extraction, transformation, and reporting Collaborate with analytics and reporting teams to deliver data-driven insights Develop scalable solutions using programming languages such as Java, Python, and Node.js Integrate APIs and third-p...

As a Data Engineer with over 7 years of experience in Data Engineering & Cloud Solutions, you specialize in data warehouse implementation, particularly with a focus on AWS services including S3, Glue, Redshift, Athena, and Lake Formation. Your skill set includes proficiency in SQL, NoSQL databases such as DynamoDB, MongoDB, Cassandra, and programming languages like Python, Spark, and Scala. Hands-on experience in real-time streaming technologies like Kafka, Kinesis, or Spark Streaming would be advantageous for this role. Possessing the AWS Certified Solutions Architect Associate certification is preferred. - Strong experience in Data Engineering & Cloud Solutions - Expertise in AWS services:...

As a Developer in this role, you will be responsible for developing analytic tools and working on Big Data and Distributed systems. Your main focus will be providing technical leadership in developing the core Analytic platform. Your expertise in innovation, problem solving, analytical skills, data structures, and design patterns will play a crucial role in the success of this position. Key Responsibilities: - Demonstrating excellence in innovation, problem solving, analytical skills, data structures, and design patterns - Leading development efforts on product features using Scala and Java - Building applications using Spark and Spark Streaming - Working with NoSQL technologies such as HBas...

As a skilled Data Engineer at BigThinkCode Technologies, your role involves designing, building, and maintaining robust data pipelines and infrastructure to optimize data flow and ensure scalability. Your technical expertise in Python, SQL, ETL/ELT frameworks, and cloud data warehouses, along with strong collaboration skills, will be crucial in partnering with cross-functional teams to enable seamless access to structured and unstructured data across the organization. **Key Responsibilities:** - Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data processing. - Optimize and manage SQL queries for performance and efficiency in handling large-scale data...

Overview This position is for Manager Data Engineering in the Commercial Data as a Service group. In this position you will enjoy being responsible for helping define and maintain the data systems key to delivering successful outcomes for our customers. will lead a software development team focused on our core services someone who combines deep technical experience with great people management skills and can help elevate our teams and our approach to engineering. We are building a culture that fosters innovation, built on a foundation of inclusion and diversity. We are looking for a candidate that has proven experience in coding, architecture leadership, and leading engineers in a fast-paced...