12 Spark Optimization Jobs

As a Senior Developer with 7 to 10 years of experience in Spark Optimization, Python, and Apache Spark, specializing in Asset and Wealth Management, your role will involve developing and optimizing data solutions to enhance asset management processes. This hybrid position does not require travel and operates during day shifts. - Develop and optimize data processing solutions using Apache Spark to improve efficiency and performance in asset management workflows. - Collaborate with cross-functional teams to design and implement scalable data architectures that support business objectives. - Analyze complex data sets to identify trends and insights that drive strategic decision-making in wealth...

Overview Join Intuit's Business Intelligence (BI) Platform team as we reimagine the next generation of scalable, intelligent data infrastructure. We serve over 240TB of data, 2 billion records daily, and deliver 200+ million report requests through 20+ complex pipelinessupporting enterprise and mid-market customers on their most critical decisions. We are seeking a Senior Data Engineer to join our Data Platform team, with a focus on designing robust data models, building scalable ETL/ELT pipelines, and enabling trustworthy, high-quality data for analytics, reporting, and intelligent systems. In this role, you will play a critical part in evolving our data architecture, ensuring data quality,...

As a candidate for this role, you would be responsible for the following: - Understanding of Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. - Experience with Spark optimization techniques. - Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning. - Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer. - Proficiency in Python/Scala/Java for Spark development and integrate with ETL process. - Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database). - Understanding of data quality best practices and data validation techniques. Qualificatio...

About The Role We are seeking an experienced Data Engineer to design, implement, and optimize a global data handling and synchronization solution across multiple regions. You will work with cloud-based databases, data lakes, and distributed systems, ensuring compliance with data residency and privacy requirements (e.g., GDPR). Requirements 6+ years of experience as a Data Engineer in cloud environments (preferably Microsoft Azure). Strong knowledge of Azure SQL, Data Lake, Data Factory, PySpark, and related services. Experience in Spark Optimization. Experience with distributed data architectures and data synchronization across regions. Familiarity with data privacy, security, and compliance...

As an experienced Data Engineer, your role will involve designing, implementing, and optimizing a global data handling and synchronization solution across multiple regions. You will work with cloud-based databases, data lakes, and distributed systems, ensuring compliance with data residency and privacy requirements such as GDPR. Key Responsibilities: - Design and implement data pipelines for global and regional data synchronization using Azure SQL, Data Lake, Data Factory, PySpark, etc. - Develop solutions for secure handling of PII and non-PII data, ensuring compliance with GDPR and other regulations. - Build and optimize ETL processes for anonymization, transformation, global synchronizati...

As an employee at Impetus Technologies, a digital engineering company, you will be part of a team dedicated to providing expert services and products to help enterprises achieve their transformation goals. Specializing in solving challenges related to analytics, AI, and cloud technologies, we empower businesses to drive innovation and growth. Your role will involve: - Working with technologies such as Bigdata, Pyspark, Hive, and Spark Optimization to develop cutting-edge solutions for our clients. Qualifications required for this position include: - Proficiency in Bigdata, Pyspark, Hive, and Spark Optimization. - Experience in working with GCP would be beneficial. About Impetus Technologies:...

As a candidate for the position, you should possess the following qualifications and skills: - Experience: 4 to 7 years - Proficiency in Bigdata, Pyspark, Hive, and Spark Optimization - Good to have: GCP Please note the Job Reference Number for this position is 13159.,

To transform data into a format that can be easily analyzed, you will be responsible for developing, maintaining, and testing infrastructures for data generation. Working closely with data scientists, you will play a key role in architecting solutions that enable data scientists to perform their tasks efficiently. Your main responsibilities will include: - Developing and maintaining scalable data pipelines and building new API integrations to support increasing data volume and complexity - Connecting offline and online data to enhance customer behavior understanding for personalization - Cleansing data, improving data quality, and preparing data for analysis - Processing data through cleansi...

We are seeking a highly skilled PySpark Developer with expertise in Distributed data processing to optimize Spark Jobs and ensure efficient data processing in a Big Data platform. This role demands a deep understanding of Spark performance tuning, distributed computing, and Big data architecture. Responsibilities include creating data tools for analytics and data scientist team members, collaborating with data and analytics experts to enhance functionality in the data system, and having the ability to adjust priorities quickly based on circumstances. The ideal candidate should have 8+ years of relevant experience in Apps Development or systems analysis. Key Responsibilities: - Analyze and co...

You will be responsible for developing, optimizing, and maintaining business intelligence and data warehouse systems. Your main accountabilities include developing scalable data pipelines, integrating offline and online data for improved customer understanding, managing data quality and governance strategies, transforming data for analysis, and ensuring data accessibility for the enterprise. To qualify for this role, you should have a Masters/Bachelors degree in Engineering/Computer Science/Math/Statistics or equivalent, strong programming skills in Python/Pyspark/SAS, experience with large datasets and technologies like Hadoop, Hive, and Spark optimization. Additionally, experience with clo...

Hiring, Data Engineer 4+ years of hands-on experience Experience in building and optimizing Big Data data pipelines, architectures, and data sets. Experience with big data tools: Hadoop, Spark Optimization, Spark-Streaming, Hive, Kafka, Hbase, Airflow, etc. Hands-on in SQL

About Impetus: Impetus Technologies is a digital engineering company dedicated to offering expert services and products to support enterprises in accomplishing their transformation objectives. Specializing in solving the analytics, AI, and cloud challenges, we empower businesses to foster unparalleled innovation and expansion. Established in 1991, we stand out as leaders in cloud and data engineering, catering cutting-edge solutions to Fortune 100 corporations. Our headquarters are located in Los Gatos, California, while our development centers span across NOIDA, Indore, Gurugram, Bengaluru, Pune, and Hyderabad, boasting a global team of over 3000 professionals. Additionally, we have operati...