18 Sparkstreaming Jobs

As an experienced professional with over 3 years of experience in Big Data & data related technologies, you possess expert level understanding of distributed computing principles. Your expertise lies in Apache Spark and you have hands-on programming skills in Python. You are proficient in Hadoop v2, Map Reduce, HDFS, and Sqoop. Additionally, you have experience in building stream-processing systems using technologies like Apache Storm or Spark-Streaming. Key Responsibilities: - Hands-on experience in Apache Spark - Proficient in programming with Python - Strong knowledge of Hadoop v2, Map Reduce, HDFS, and Sqoop - Building stream-processing systems using Apache Storm or Spark-Streaming - Wor...

Title- Pyspark Developer Job Description Must-Have Minimum 6-8years of experience in build & deployment of Bigdata applications using SparkSQL, SparkStreaming in Python; Minimum 5 years of extensive experience in design, build and deployment of Python-based applications; Design and develop ETL integration patterns using Python on Spark. Develop framework for converting existing PowerCenter mappings and to PySpark(Python and Spark) Jobs. Create Pyspark frame to bring data from multiple databases to Amazon S3. Expertise on graph algorithms and advanced recursion techniques. Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses. Good-to-Have Hands-on expe...

As an Azure Databricks+ MS Fabrics+ Python Data Engineer at PWC AC, you will play a crucial role in designing and implementing scalable ETL/ELT pipelines using Databricks and Apache Spark. Your responsibilities will include: - Proficiency with Azure Data Services, Databricks, and Snowflake - Deep understanding of traditional and modern data architecture and processing concepts - Hands-on experience with Snowflake utilities, SnowSQL, SnowPipe, ETL data pipelines, and Big Data model techniques using Python / Java - Building stream-processing systems using solutions such as Storm or Spark-Streaming - Experience with Big Data ML toolkits like Mahout, SparkML, or H2O - Knowledge in Python and cer...

Role Overview: At PwC, as a data and analytics engineering professional, you will leverage advanced technologies and techniques to design and develop robust data solutions for clients. Your primary role will involve transforming raw data into actionable insights, enabling informed decision-making, and driving business growth. In the field of data engineering at PwC, your focus will be on designing and building data infrastructure and systems to facilitate efficient data processing and analysis. You will be tasked with the development and implementation of data pipelines, data integration, and data transformation solutions. Key Responsibilities: - Possess 2+ years of experience working in Azu...

In this role at PwC, you will be part of the data and analytics engineering team, focusing on utilizing advanced technologies to develop robust data solutions for clients. Your main responsibility will involve transforming raw data into actionable insights, facilitating informed decision-making and driving business growth. Specifically in data engineering at PwC, you will concentrate on designing and constructing data infrastructure and systems to enable efficient data processing and analysis. Your duties will also include developing and implementing data pipelines, data integration, and data transformation solutions. Key Responsibilities: - Interact with Offshore Manager/Onsite Business Ana...

As a Spark Cluster Optimization Specialist at the company, your role will involve optimizing Spark clusters for cost, efficiency, and performance. Your responsibilities will include: - Implementing robust monitoring systems to identify bottlenecks using data and metrics - Providing actionable recommendations for continuous improvement - Optimizing the infrastructure required for extraction, transformation, and loading of data from various sources using SQL and AWS big data technologies - Collaborating with data and analytics experts to enhance cost efficiencies in data systems To excel in this role, you should possess the following qualifications: - Experience processing large workloads and ...

As a Big Data Engineer, you will be responsible for working on collecting, storing, processing, and analyzing huge sets of data. Your primary focus will be on choosing optimal solutions for these purposes, then maintaining, implementing, and monitoring them. Additionally, you will integrate these solutions with the architecture used across the company. Key Responsibilities: - 6+ years" experience in Big Data Hadoop, MapReduce, Hive, Sqoop, and Spark with hands-on expertise in designing and implementing high data volume solutions (ETL & Streaming). - Experience in building stream-processing systems using solutions like Storm, Spark-Streaming, and Kafka streams. - Extensive experience working ...

Role Overview: As an AWS Senior Developer at PwC - AC, you will collaborate with the Offshore Manager and Onsite Business Analyst to comprehend the requirements and take charge of implementing end-to-end Cloud data engineering solutions such as Enterprise Data Lake and Data hub in AWS. Your role will involve showcasing strong proficiency in AWS cloud technology, exceptional planning and organizational skills, and the ability to work as a cloud developer/lead in an agile team to deliver automated cloud solutions. Key Responsibilities: - Proficient in Azure Data Services, Databricks, and Snowflake - Deep understanding of traditional and modern data architecture and processing concepts - Profic...

The position is for a Full Time job with rotational shifts based in Nagpur, Pune, or Bangalore. We are looking to fill 4 positions with candidates who have 5 to 8 years of experience. As an AWS Data Engineer, you will be responsible for leading development activities for the Data engineering team. You will collaborate with other teams such as application management and product delivery, working closely with technical leads, product managers, and support teams. Your role will involve providing guidance to the development, support, and product delivery teams. Additionally, you will lead the implementation of tools and technologies to drive cost-efficient architecture and infrastructure. On the...

At PwC, the focus in data and analytics engineering is on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. You play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth. In data engineering at PwC, you will concentrate on designing and building data infrastructure and systems to enable efficient data processing and analysis. Your responsibilities include developing and implementing data pipelines, data integration, and data transformation solutions. As an AWS Architect / Manager at PwC - AC, you will interact with Offshore Manager/Onsite Business Analyst to under...

As a part of the data and analytics engineering team at PwC, your focus will be on utilizing advanced technologies and techniques to create robust data solutions for clients. Your role will involve transforming raw data into actionable insights, enabling informed decision-making, and contributing to business growth. Specifically in data engineering at PwC, you will be responsible for designing and constructing data infrastructure and systems that facilitate efficient data processing and analysis. This will include the development and implementation of data pipelines, data integration, and data transformation solutions. At PwC - AC, we are seeking an Azure Manager specializing in Data & AI, w...

You have a great opportunity to join as a Data Software Engineer with 5-12 years of experience in Big Data & Data related technology. We are looking for candidates with an expert level understanding of distributed computing principles and hands-on experience in Apache Spark along with proficiency in Python. You should also have experience with technologies like Hadoop, Map Reduce, HDFS, Sqoop, Apache Storm, Spark-Streaming, Kafka, Hive, Impala, and integration of data from various sources such as RDBMS, ERP, and Files. Additionally, knowledge of NoSQL databases, ETL techniques, SQL queries, joins, stored procedures, relational schemas, and performance tuning of Spark Jobs is required. Moreov...

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, a...

As a Data Software Engineer at KG Invicta Services Pvt Ltd, you will leverage your 5-12 years of experience in Big Data & Data-related technologies to drive impactful solutions. Your expertise in distributed computing principles and Apache Spark, coupled with hands-on programming skills in Python, will be instrumental in designing and implementing efficient Big Data solutions. You will demonstrate proficiency in a variety of tools and technologies including Hadoop v2, Map Reduce, HDFS, Sqoop, Apache Storm, Spark-Streaming, Kafka, RabbitMQ, Hive, Impala, and NoSQL databases such as HBase, Cassandra, and MongoDB. Your ability to integrate data from diverse sources like RDBMS, ERP, and files, a...

As a Data Software Engineer, you will be responsible for utilizing your 5-12 years of experience in Big Data & Data-related technologies to contribute to the success of projects in Chennai and Coimbatore in a Hybrid work mode. You should possess an expert level understanding of distributed computing principles and a strong knowledge of Apache Spark, with hands-on programming skills in Python. Your role will involve working with technologies such as Hadoop v2, Map Reduce, HDFS, Sqoop, Apache Storm, and Spark-Streaming to build stream-processing systems. You should have a good grasp of Big Data querying tools like Hive and Impala, as well as experience in integrating data from various sources ...

You should have 5-12 years of experience in Big Data & Data related technologies. Your expertise should include a deep understanding of distributed computing principles and strong knowledge of Apache Spark. Proficiency in Python programming is required, along with experience using technologies such as Hadoop v2, Map Reduce, HDFS, Sqoop, Apache Storm, and Spark-Streaming for building stream-processing systems. You should have a good understanding of Big Data querying tools like Hive and Impala, as well as experience in integrating data from various sources such as RDBMS, ERP, and Files. Knowledge of SQL queries, joins, stored procedures, and relational schemas is essential. Experience with No...

You should have 5-12 years of experience in Big Data & Data related technologies, with expertise in distributed computing principles. Your skills should include an expert level understanding of Apache Spark and hands-on programming with Python. Proficiency in Hadoop v2, Map Reduce, HDFS, and Sqoop is required. Experience in building stream-processing systems using technologies like Apache Storm or Spark-Streaming, as well as working with messaging systems such as Kafka or RabbitMQ, will be beneficial. A good understanding of Big Data querying tools like Hive and Impala, along with integration of data from multiple sources including RDBMS, ERP, and Files, is necessary. You should possess know...

You will be part of our Data Engineering Team in Gurgaon, Noida, or Pune, contributing to the development and management of large enterprise data and analytics platforms. Your role will involve collaborating with the data engineering and data science teams to implement scalable data lakes, data ingestion platforms, machine learning analytics platforms, and more. With at least 3 years of industry experience, you will be responsible for creating end-to-end data solutions and optimal data processing pipelines for handling large volumes of diverse data types. Proficiency in Python, including knowledge of design patterns and strong design skills, is essential. You should have expertise in working...