210 Spark Streaming Jobs - Page 2

As a skilled Data Engineer at BigThinkCode Technologies, your role involves designing, building, and maintaining robust data pipelines and infrastructure to optimize data flow and ensure scalability. Your technical expertise in Python, SQL, ETL/ELT frameworks, and cloud data warehouses, along with strong collaboration skills, will be crucial in partnering with cross-functional teams to enable seamless access to structured and unstructured data across the organization. **Key Responsibilities:** - Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data processing. - Optimize and manage SQL queries for performance and efficiency in handling large-scale data...

Overview This position is for Manager Data Engineering in the Commercial Data as a Service group. In this position you will enjoy being responsible for helping define and maintain the data systems key to delivering successful outcomes for our customers. will lead a software development team focused on our core services someone who combines deep technical experience with great people management skills and can help elevate our teams and our approach to engineering. We are building a culture that fosters innovation, built on a foundation of inclusion and diversity. We are looking for a candidate that has proven experience in coding, architecture leadership, and leading engineers in a fast-paced...

As a Senior Azure Data Engineer, you should have a minimum experience in working on projects involving Azure Data Engineering. Your responsibilities will include: - Analyzing data using Python, Spark streaming, SSIS/Informatica Batch ETL, and database tools like SQL, Mongo DB for processing data from various sources. - Designing, operating, and integrating different systems to enhance efficiencies in key business areas. - Understanding business requirements, interacting with business users, and reverse engineering existing data products. - Having a good understanding and working knowledge of distributed databases and pipelines. - Ability to analyze and identify the root cause of technical is...

As a Senior Data Engineer at Barclays, you will play a crucial role in building and maintaining the systems that collect, store, process, and analyze data. Your responsibilities will include developing data pipelines, data warehouses, and data lakes to ensure accuracy, accessibility, and security of all data. **Key Responsibilities:** - Hands-on experience working with large-scale data platforms and developing cloud solutions in AWS data platform to drive business success. - Strong understanding of AWS and distributed computing paradigms, designing and developing data ingestion programs using Glue, Lambda, S3, Redshift, Snowflake, and Databricks. - Developing data ingestion programs to proce...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

JOB DESCRIPTION Data Engineer Designation Data Engineer Experience 5+ Years Location Mumbai (onsite) Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: • Design, develop, and maintain real-time and batch data pipelines in Databricks. • Integrate Apache Kafka with Databricks using Structured Streaming. • Implement robust ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

As a member of the Managed Services team at PwC, you will focus on providing outsourced solutions and supporting clients across various functions. Your role will involve streamlining operations, reducing costs, and enhancing efficiency by managing key processes and functions on behalf of organizations. You will leverage your skills in project management, technology, and process optimization to deliver high-quality services to clients. Specifically, in managed service management and strategy, your responsibilities will include transitioning and running services, managing delivery teams, programmes, commercials, performance, and delivery risk. Your work will revolve around continuous improveme...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Role Overview: At PwC, your focus in managed services will involve providing outsourced solutions and supporting clients across various functions. You will assist organisations in streamlining operations, reducing costs, and enhancing efficiency by managing key processes and functions on their behalf. Your expertise in project management, technology, and process optimization will enable you to deliver high-quality services to clients. As part of the managed service management and strategy team at PwC, you will be responsible for transitioning and running services, managing delivery teams, programmes, commercials, performance, and delivery risk. Your role will revolve around continuous improv...

As a Data Engineer specializing in Spark & Delta Lake, your role will involve understanding key Spark concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You will also be expected to have experience in optimizing Spark performance and a deep understanding of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities will include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python, Scala, or Java for Spark development and integration with ETL processes is essential. Additionally, you should possess knowledge of data ingestion techniques from various sources such ...

Role & responsibilities Position: Senior Developer (Azure Data Engineer) Experience: 6-8 Years No. Of Position: 2 Description: At PwC, were always looking to strengthen our business by adding the best available people to our team. We’re currently seeking a Senior Azure Data Engineer to work closely with our Data Analytics team in driving the adoption, optimization, and iteration, leveraging multiple integrations and technologies. The ideal candidate will have at least 6-10 years of experience with technical implementation of customer projects. Using their expertise, problem-solving ability, and creative flair, the consultant will design, build, deliver, and maintain. The consultant will also...

Streaming data Technical skills requirements :- Experience- 5+ Years Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred) - Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR - Hands-on experience of programming language like Scala with Spark. - Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases - Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred - Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie - Hands on working experience of data processing at scale with event driven syste...

Role Overview: You will be joining the ICG TTS Operations Technology (OpsTech) Group to assist in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role as a Lead Spark Scala Engineer will involve managing development teams in the distributed systems Eco-System and collaborating effectively as a team player. Your primary objective will be to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical leadership and guidance to a team of ...

Title: Data Engineer (Neo4j, JanusGraph) Location: Bangalore Look for Exp. between 3-5 years - Must have graph databases (e.g., Neo4j, JanusGraph) and graph query languages - its must. Look for batch and real time streaming exp. Look with Kafka OR Flink Exp. Data structures/algorithms Hadoop ecosystems such as Hive, Iceberg, Spark SQL 1 coding language is must - (Java or Python or Scala) Job Description: The Cloud Data Technologies (CDT) team at eBay oversees data infrastructure and the management of the end-to-end data lifecycle. As a Data Engineer on our team, you will work on our Hadoop-based data warehouse, contributing to scalable and reliable big data solutions for analytics and busine...

Exp 5 to 7yrs Location – Mumbai Responsibilities:• Perform data enquiry, data analysis and data sourcing in Relational and Big Data database environments.• Develop a thorough understanding of the data definitions, domain values, data relationships, business rules, sources, and data integration used in the business domain• Develop a thorough understanding of the technology systems; data stores, data pipelines, job streams etc.• Translate business requirements into data models and program design specifications• Modify ETL and other programs to implement changes to business requirements• Respond to questions from business analysts and developers regarding query building and optimization• Troubl...

We are looking for a skilled professional with expertise in Spark and Scala to join our team. The ideal candidate will have 4 to 10 years of experience. Roles and Responsibility Design, develop, and implement scalable data processing pipelines using Spark and Scala. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems using Big Data Technologies such as Hadoop, Hive, and HBase. Troubleshoot and optimize performance issues in existing data systems. Participate in code reviews and ensure adherence to coding standards. Stay updated with industry trends and emerging technologies in the field of Spark and Scala. Job...

Spark & Delta Lake Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL and Spark Streaming. Experience with Spark optimization techniques. Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database) Understanding of data quality best practices and data validation techniques. Other Skills: Understanding of data warehouse concepts, data modelling techniques. ...

Extensive coding expand knowledge n Event driven & streaming architecture Experience in JVM tuning for performance Knowledge on Containerisation using Docker and Kubernetes. Wrking knowledge on Caching systems, with particular exp using Apache Ignite

Role & responsibilities Job Description We are looking for a Senior Java Developer with a strong background in Java , Spring Framework , Spark Batch , and Spark Streaming to join our dynamic and growing team. You will be responsible for designing and developing high-performance, scalable applications for batch and real-time data processing . Key Responsibilities Develop and maintain Java-based applications using Spring Boot , Spring MVC , or Spring Data . Design and implement batch data processing solutions using Apache Spark (Batch) . Build robust real-time data pipelines using Spark Streaming . Collaborate with product owners and cross-functional teams to design, develop, and deliver new f...

Skills: Complex Databricks implementations,Architect level certification from Databricks, Python, Scala and SQL,Strong depth in Lakehouse Architecture with Delta Tables, Expertise in Spark ecosystems, Data frame API, spark streaming, Data set API, RDD APIs & Spark SQL.,Expertise Notice Period: Immediate -45 days

Role Overview: You will be working as a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group to help in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role will involve managing development teams in the distributed systems Eco-System and being a strong team player. It is expected that you will have superior technical knowledge of current programming languages, technologies, and other leading-edge development tools to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal perf...

Role Overview: You will be working as a Solutions Architect at Koantek, collaborating with customers to design scalable data architectures using Databricks technology and services. Your role will involve building secure, highly scalable big data solutions, focusing on achieving tangible, data-driven outcomes while ensuring simplicity and operational effectiveness. With your technical expertise and business acumen, you will engage in complex technology discussions, demonstrating the value of the Databricks platform throughout the sales process. Collaborating with Account Executives, you will interact with customers" technical leaders to become a trusted advisor who delivers concrete results. ...

Role Overview: At PwC, as a Data Engineer in the offshore managed services team, you will be responsible for designing, implementing, and maintaining scalable data pipelines and systems that support data-driven initiatives. Your role will involve working with a variety of data technologies such as Teradata, DataStage, AWS, Databricks, SQL, and more to streamline operations, reduce costs, and improve efficiency for clients. Key Responsibilities: - Design, develop, and maintain scalable ETL pipelines using DataStage and other ETL tools. - Leverage AWS cloud services for data storage, processing, and analytics. - Utilize Databricks to analyze, process, and transform data, ensuring high performa...

As an Applications Development Programmer Analyst at our company, your role will involve participating in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team. Your main objective will be to contribute to applications systems analysis and programming activities. **Key Responsibilities:** - Utilize your knowledge of applications development procedures and concepts to identify and define necessary system enhancements. - Identify and analyze issues, make recommendations, and implement solutions based on your understanding of business processes, system processes, and industry standards to solve complex issues. - Analyze ...