279 Spark Streaming Jobs - Page 2

Job Summary The Big Data Engineer is responsible for designing, developing, and maintaining large-scale data processing systems. The role includes building efficient data pipelines, working with distributed technologies, and ensuring data availability, reliability, and performance for analytics and business use cases. Key Responsibilities 1. Data Pipeline Development Build scalable ETL/ELT pipelines for ingesting and transforming large datasets. Develop batch and real-time data processing solutions using Apache Spark, Kafka, Hive, Flink , or similar tools. Optimize data workflows for performance and cost. 2. Big Data Ecosystem Ownership Work with Hadoop ecosystem components (HDFS, Hive, HBas...

As a Spark Scala Architect, you will play a crucial role in designing, developing, and optimizing Spark solutions for large-scale implementations on cloud platforms. Your expertise in Spark & Scala, Spark Streaming, HDFS, HBase, Hive, and MapReduce will be essential for analyzing, designing, and developing Spark batch job processing and Spark streaming applications. Your responsibilities will include: - Minimum 10-12 years of hands-on experience in Spark Scala, with a strong emphasis on Spark Streaming - Proficiency in designing, developing, debugging, and performance tuning of Scala code - In-depth knowledge of architecture concepts and the ability to propose and implement new ideas - Stron...

The Data Engineering team is responsible for building and maintaining the data applications and platform that enables the ISS business to operate. This role is appropriate for a senior data engineer capable of taking ownership and a delivering a subsection of the wider data platform or data applications. As a Senior Data Engineer, you will directly contribute to our key organizational objectives of Accelerated Innovation, Cost Optimization, Risk Mitigation, and Business Enablement Key Responsibilities • Implement and maintain data pipelines for ingestion, transformation, and delivery • Ensure data quality through validation and monitoring processes • Collaborate with senior engineers to desi...

Work Location: Pan India Experience: 13+yrs Required Skills: AWS Bigdata engineer, AWS Cloud, PySpark (Python + Spark), EMR, Lambda, S3, CICD, Airflow Thanks & Regards Suganya R suganya@spstaffing.in

Role Overview: You will be responsible for developing and maintaining Java applications, REST APIs, and Spring Boot services. Additionally, you will be required to work on front-end technologies such as Angular JS, React JS, or Vue JS. Your role will involve handling data cleaning, visualization, and reporting tasks while ensuring smooth operations in an agile environment. Your strong analytical and problem-solving skills will be crucial in this role, along with effective communication and interpersonal abilities. Key Responsibilities: - Develop and maintain Java applications, REST APIs, and Spring Boot services. - Work on front-end technologies like Angular JS, React JS, or Vue JS. - Handle...

We are seeking an experienced Apache Spark Developer to design, develop, and optimize big data processing pipelines using Spark . The ideal candidate will have strong expertise in distributed computing , data engineering , and performance tuning , with experience in building scalable data solutions on cloud or on-prem platforms. Key Responsibilities: Design and implement data processing pipelines using Apache Spark (Core, SQL, Streaming). Develop and optimize ETL workflows for large-scale data ingestion and transformation. Work with Spark on Hadoop , Databricks , or cloud-based Spark environments . Collaborate with data engineers , data scientists , and business teams to deliver analytics so...

Primary Responsibilities: Analyze business requirements & functional specifications Be able to determine the impact of changes in current functionality of the system Interaction with diverse Business Partners and Technical Workgroups Be flexible to collaborate with onshore business, during US business hours Be flexible to support project releases, during US business hours Required Qualifications: Undergraduate degree or equivalent experience 3+ years of working experience in Python, Pyspark, Scala 3+ years of experience working on MS Sql Server and NoSQL DBs like Cassandra, etc. Hands-on working experience in Azure Databricks Solid healthcare domain knowledge Exposure to following DevOps met...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

HIRINGS FOR SPARK-SCALA-PYTHON DEVELOPER Responsibilities : As part of the Infosys delivery team, your primary role would be to interface with the client for quality assurance, issue resolution and ensuring high customer satisfaction. You will understand requirements, create and review designs, validate the architecture and ensure high levels of service offerings to clients in the technology domain. You will participate in project estimation, provide inputs for solution delivery, conduct technical risk planning, perform code reviews and unit test plan reviews. You will lead and guide your teams towards developing optimized high quality code deliverables, continual knowledge management and ad...

As a Manager in the EY-GDS Data and Analytics team, you will be responsible for designing and proposing Azure-based data and AI solutions that align with customer objectives and solve business challenges. You will have the opportunity to architect and implement data platforms supporting real-time, batch, and hybrid analytics. Additionally, you will collaborate with cross-functional teams to ensure data security, scalability, and performance. Your key responsibilities will include the following: - Designing and proposing Azure-based data and AI solutions - Architecting and implementing data platforms for various analytics - Recommending design alternatives for data ingestion, processing, and ...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

HIRINGS FOR SPARK DEVELOPER Responsibilities : As part of the Infosys delivery team, your primary role would be to interface with the client for quality assurance, issue resolution and ensuring high customer satisfaction. You will understand requirements, create and review designs, validate the architecture and ensure high levels of service offerings to clients in the technology domain. You will participate in project estimation, provide inputs for solution delivery, conduct technical risk planning, perform code reviews and unit test plan reviews. You will lead and guide your teams towards developing optimized high quality code deliverables, continual knowledge management and adherence to th...

Technical Skill- Required: Advanced knowledge of the principles of computer application including software engineering, system analysis and design, design and analysis of algorithms, digital systems, computer system architecture, data structures, computer networks, database management system, and problem solving and computer programming, among others. Core hands on development skills in Bigdata ecosystem i.e. Spark Scala, SOLR, Pig, Hive, Kafka and HBase. Experience with real-time data delivery, Kafka, NiFi, and Change-data-capture Optimize and tune queries, job flows on Hadoop environments to meet performance requirements. Good understanding of release management, CI/CD Work with Bigdata de...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

We are seeking an experienced Apache Spark Developer to design, develop, and optimize big data processing pipelines using Spark . The ideal candidate will have strong expertise in distributed computing , data engineering , and performance tuning , with experience in building scalable data solutions on cloud or on-prem platforms. Key Responsibilities: Design and implement data processing pipelines using Apache Spark (Core, SQL, Streaming). Develop and optimize ETL workflows for large-scale data ingestion and transformation. Work with Spark on Hadoop , Databricks , or cloud-based Spark environments . Collaborate with data engineers , data scientists , and business teams to deliver analytics so...

Knowledge of Pyspark, Big Data experience, AWS EMR, S3, IAM, Lambda, SNS, SQS Required Candidate profile AWS, Terraform, Pyspark, Java Spark with Kafka streaming, Scala, Databricks

We are currently considering only candidates based in Bangalore who can attend in-person interviews after CV shortlisting. We're open to considering candidates with 5 to 12 years of experience, provided you are a hands-on coder with strong technical skills. The Modern Data Engineer is responsible for designing, implementing, and maintaining scalable data architectures using cloud technologies, primarily on AWS, to support the next evolutionary stage of the Investment Process. They build robust data pipelines, optimize data storage, and access patterns, and ensure data quality while collaborating across engineering teams to deliver high-value data products Key Responsibilities • Implement and...

We are seeking a highly skilled and experienced Senior Data Engineer (Azure+DBT) with strong expertise in Streaming Data and Big Data Technologies . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Strong expertise in: Azure Cloud Services o Databricks (including workflows and notebooks) PySpark and Python o SQL (complex queries, performance tuning) Azure Data Factory (ADF) DBT (Data Build Tool) o ServiceNow (incident and change management) Experience in reverse engineering and legacy system modernization is a plus. Excellent problem-solving and communication skills.

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

Must have minimum of 12 15 years of experience implementing batch and streaming applications using Spark and Scala Experience with Confluent or Apache Kafka and Connectors Experience with Java and building microservices, web applications or other applications Develop data pipelines responsible for ingesting large amounts of data from various sources Help evolve data architecture and work on Next Generation real time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems Write code and develop worker node for business logic, ETL and orchestration processes Develop algorithms for better attribution rules and category classifiers Work...