905 Apache Spark Jobs - Page 18

As an AWS Senior Data Engineer at EY, you will play a crucial role in designing, developing, and maintaining scalable data pipelines for data ingestion and processing using Python, Spark, and various AWS services. With a minimum of 5+ years of experience, you will be responsible for ensuring data quality, optimizing data storage solutions, and collaborating with cross-functional teams to deliver solutions that meet business needs. Your key responsibilities will include working with on-prem Oracle databases, batch files, and Confluent Kafka for data sourcing, implementing ETL processes using AWS Glue and EMR for batch and streaming data, and developing data storage solutions using Medallion A...

Notice Period : Immediate to 30 days Mandatory Scala and Python Apache Spark (batch & streaming) must! Deep knowledge of HDFS internals and migration strategies. Experience with Apache Iceberg (or similar table formats like Delta Lake / Apache Hudi) for schema evolution, ACID transactions, and time travel. Running Spark and/or Flink jobs on Kubernetes (e.g., Spark-on-K8s operator, Flink-on-K8s). Experience with distributed blob storages like Ceph or AWS S3 and similar Building ingestion, transformation, and enrichment pipelines for large-scale datasets. Infrastructure-as-Code (Terraform, Helm) for provisioning data infrastructure. Ability to work independently while guiding juniors. Nice to ...

Your opportunity to make a real impact and shape the future of financial services is waiting for you. Let's push the boundaries of what's possible together. As a Senior Director of Software Engineering at JPMorgan Chase within the Consumer and Community Banking division, you will be responsible for leading various technical domains, overseeing the activities of multiple departments, and fostering cross-functional collaboration. Your technical expertise will be utilized across different teams to promote the adoption and implementation of advanced technical methods, helping the firm stay ahead of industry trends, best practices, and technological advancements. Leads multiple technology and pro...

As a senior architect cum development lead at UBS, your primary role will involve rebuilding the Credit Risk Management solution by reimagining workflows, redesigning data layout, and transitioning the application to a cloud-native platform to provide near real-time updates. Your focus will be on delivering quality solutions that enhance productivity, support business expansion, and ensure user satisfaction. Credit Risk Officers require enhanced automation and intelligence within the applications to effectively assess risks, offering you the opportunity to shape the technical future and take ownership of Cloud Engineering by architecting common components and services. Your responsibilities ...

As a Technical Lead/Manager in the AI Center of Excellence (CoE) team at our organization, you will be responsible for leading the team and driving the implementation of AI-powered solutions. Your role will involve coordinating with multiple teams, collaborating closely with stakeholders, and overseeing the technical aspects of AI/ML projects to ensure the effective application of generative AI technologies in our product suite. With 6-8 years of experience in AI/ML technologies, including at least 3 years in a leadership role, you will possess a strong background in AI/ML model development and deployment, especially within the financial services sector. Your proficiency in Python, R, and ot...

Role & responsibilities Data Engineer role is to design, build, and maintain scalable data pipelines and infrastructure that ensure reliable, efficient, and secure data flow across systems to support analytics, reporting, and decision-making. Skills : Python, SQL, RESTful APIs, Databricks, Apache Spark, Cloud Platform (Azure)

Optum is a global organization dedicated to delivering care and utilizing technology to enhance the well-being of millions of individuals. When you join our team, your work will directly contribute to improving health outcomes by facilitating access to essential care, pharmacy benefits, data, and resources that promote optimal health. Our workplace fosters a culture of diversity and inclusion, where you will collaborate with talented colleagues, avail comprehensive benefits, and have opportunities for career growth. Embrace the opportunity to impact the communities we serve and contribute to advancing health equity on a global scale. Start your journey with us by caring, connecting, and grow...

As a Data bricks Architect at Gramener, you will have the opportunity to engage in diverse impactful customer technical Big Data projects. Your responsibilities will include developing reference architectures, how-to guides, and minimally viable products (MVPs). Leading strategic initiatives encompassing the end-to-end design, build, and deployment of industry-leading big data and AI applications will be a key aspect of your role. You will provide architectural guidance to foster the adoption of Databricks across business-facing functions and collaborate with platform engineering teams to implement Databrick services effectively within the infrastructure. Your expertise in AWS administration...

As a Data Engineer with 5 to 8 years of experience, you will be responsible for working with Azure Data Engineering platforms such as Azure Data Factory, Azure Databricks, and Azure Key Vault. Your expertise in Big Data technologies, particularly Apache Spark, will be crucial for this role. Additionally, a deep understanding of Azure databases and storage solutions like Azure SQL Database and Azure Data Lake Storage is required. Your primary focus will be on developing end-to-end ETL pipelines and utilizing your strong SQL skills to ensure efficient data processing. If you are ready to take on a new challenge and possess the must-have skills mentioned above, we invite you to connect with us ...

As a Data Engineer II at Media.net, you will be responsible for designing, executing, and managing large and complex distributed data systems. Your role will involve monitoring performance, optimizing existing projects, and researching and integrating Big Data tools and frameworks as required to meet business and data requirements. You will play a key part in implementing scalable solutions, creating reusable components and data tools, and collaborating with teams across the company to integrate with the data platform efficiently. The team you will be a part of ensures that every web page view is seamlessly processed through high-scale services, handling a large volume of requests across 5 m...

The Applications Development Senior Programmer Analyst position is an intermediate level role where you will be responsible for contributing to the establishment and implementation of new or updated application systems and programs in collaboration with the Technology team. Your main objective will be to participate in applications systems analysis and programming activities. Responsibilities: - Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas. - Monitor and control all phase...

As a Databricks AWS/Azure/GCP Architect at Koantek based in Mumbai, you will play a crucial role in building secure and highly scalable big data solutions that drive tangible, data-driven outcomes while emphasizing simplicity and operational efficiency. Collaborating with teammates, product teams, and cross-functional project teams, you will lead the adoption and integration of the Databricks Lakehouse Platform into the enterprise ecosystem and AWS/Azure/GCP architecture. Your responsibilities will include implementing securely architected big data solutions that are operationally reliable, performant, and aligned with strategic initiatives. Your expertise should include an expert-level know...

As a Data Engineer, you will be responsible for designing, developing, and optimizing data pipelines and ETL workflows using AWS Glue, AWS Lambda, and Apache Spark. Your role will involve implementing big data processing solutions utilizing AWS EMR and AWS Redshift. You will also be tasked with developing and maintaining data lakes and data warehouses on AWS, including S3, Redshift, and RDS. Ensuring data quality, integrity, and governance will be a key aspect of your responsibilities, which will be achieved through leveraging AWS Glue Data Catalog and AWS Lake Formation. It will be essential for you to optimize data storage and processing for both performance and cost efficiency. Working wi...

Databricks Developer (5–7 yrs exp) skilled in Apache Spark, Design & Implementation, PySpark, Delta Lake, and cloud platforms (Azure/AWS). Must excel in data pipelines, ETL, governance, and performance tuning. Unity Catalog, ML, and CI/CD experience.

Skill: Databricks Production Support - Senior Level Experience: 8-12 Years Location: Hyderabad & Bangalore Notice Period: Immediate - 15 Days Shift Timings: 1PM - 10PM Detailed job description - Skill Set: We are seeking a highly skilled Databricks Platform Operations to join our team, responsible for daily monitoring and resolution of data load issues, platform optimization, capacity planning, and governance management. This role is pivotal in ensuring the stability, scalability, and security of our Databricks environment while acting as a technical architect for platform best practices. The ideal candidate will bring a strong operational background, potentially with earlier experience as a...

Skill: Databricks Production Support - Senior Level Experience: 8-12 Years Location: Hyderabad & Bangalore Notice Period: Immediate - 15 Days Shift Timings: 1PM - 10PM Detailed job description - Skill Set: We are seeking a highly skilled Databricks Platform Operations to join our team, responsible for daily monitoring and resolution of data load issues, platform optimization, capacity planning, and governance management. This role is pivotal in ensuring the stability, scalability, and security of our Databricks environment while acting as a technical architect for platform best practices. The ideal candidate will bring a strong operational background, potentially with earlier experience as a...

• Strong programming skills in Python and SQL. • Hands-on experience with Databricks and Apache Spark. • Proficiency in working with RESTful APIs and data integration techniques•Experience CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOp

As a Python Developer specialized in Artificial Intelligence (AI) and Machine Learning (ML), you will play a crucial role in designing, developing, and deploying advanced machine learning models and AI-based solutions. In this position in Bangalore, with a required experience of 4 years, you will collaborate closely with data scientists and other developers to implement cutting-edge algorithms and data-driven insights that add significant business value. Your key responsibilities will include designing and maintaining machine learning models using Python, with a focus on ensuring optimal performance and scalability. You will be tasked with implementing algorithms for various AI-driven applic...

The Applications Development Senior Programmer Analyst position is an intermediate level role that involves participating in the establishment and implementation of new or revised application systems and programs in collaboration with the Technology team. The primary goal of this position is to contribute to applications systems analysis and programming activities. You should have a minimum of 10 years of experience in application/software development and maintenance, with a specific requirement of experience in the Banking domain. Additionally, you must possess at least 8 years of experience in Big Data Technologies like Apache Spark, Hive, and Hadoop. It is essential to have worked with Ab...

The Universal Data Catalog (UDC) / Visa Data Catalog (VDC) is a comprehensive metadata management platform designed to provide a centralized repository for all data-related information across the organization. The platform enables efficient data discovery, governance, and utilization by offering detailed metadata definitions for tables, columns, and other data assets. It supports various business units by improving data accessibility, enhancing data quality, and facilitating compliance with data governance policies. Our implementation leverages the open-source Datahub project, ensuring a robust, flexible, and scalable solution. Key Responsibilities Develop and maintain the Enterprise Data Ca...

As a Data Engineer, you will be responsible for designing and implementing next-generation data pipelines, analytics, and Business Intelligence solutions. You will manage various AWS resources such as RDS, EMR, MWAA, and Lambda. Your primary focus will be on developing high-quality data architecture and pipelines to support business analysts and data scientists. Collaborating with other technology teams, you will extract, transform, and load data from diverse sources. Additionally, you will work towards enhancing reporting and analysis processes by automating self-service support for customers. You should have a solid background with 7-10 years of data engineering experience, including exper...

As a Senior Data Scientist at CDM Smith, you will play a crucial role in our Digital Engineering Solutions team, specifically within the Data Technology group. This position involves contributing to strategic initiatives in Architecture, Engineering, and Construction (AEC) by leveraging cutting-edge data technologies and analytics. You will be instrumental in providing actionable business insights and developing robust solutions that cater to the needs of AEC professionals and client outcomes. Your responsibilities will include developing and implementing predictive, prescriptive, and advanced analytics models using various statistical, optimization, and machine learning techniques to addres...

The Natural Language Query (NLQ) platform is an innovative initiative designed to transform the way users interact with data. Our platform leverages advanced natural language processing (NLP) to convert user queries in plain language into executable SQL queries, enabling seamless data retrieval and analysis without the need for SQL expertise. The NLQ platform will be powered by the Enterprise Data Catalog, ensuring comprehensive and accurate metadata definitions for all table and column information. This platform empowers different business units to build external-facing conversational BI chatbots to handle customer requests, while also significantly reducing data exploration efforts for int...

Your opportunity to make a real impact and shape the future of financial services is waiting for you. Let's push the boundaries of what's possible together. As a Senior Director of Software Engineering at JPMorgan Chase within the Consumer and Community Banking division, you will be responsible for leading various technical domains, overseeing the activities of multiple departments, and fostering cross-functional collaboration. Your technical expertise will be utilized across different teams to promote the adoption and implementation of advanced technical methods, helping the firm stay ahead of industry trends, best practices, and technological advancements. Leads multiple technology and pro...

Role & responsibilities Key Responsibilities Design, develop, and maintain scalable big data pipelines for both batch and real-time data ingestion. Build and optimize stream-processing applications using Apache Flink, Kafka Streams, or similar frameworks. Work on distributed data technologies such as Hadoop, MapReduce, Spark, and Kafka to handle large-scale data processing. Implement best practices in data engineering, ensuring performance, reliability, and security. Collaborate with cross-functional teams to support data-driven applications and analytics use cases . Leverage cloud platforms (AWS, Azure, GCP) to deploy and scale data solutions. Key Skills Required Strong Software Engineering...