876 Apache Spark Jobs - Page 4

As a versatile and highly skilled Data Analyst / AI Engineer, you will have the opportunity to combine the strengths of a data scientist and an AI engineer. Your role will involve diving deep into data, extracting meaningful insights, and building and deploying cutting-edge Machine Learning, Deep Learning, and Generative AI models. You will be instrumental in transforming raw data into strategic assets and intelligent applications. **Key Responsibilities:** - **Data Analysis & Insight Generation:** - Perform in-depth Exploratory Data Analysis (EDA) to identify trends, patterns, and anomalies in complex datasets. - Clean, transform, and prepare data from various sources for analysis and model...

As a Data Engineering Manager at our company, you will play a crucial role in leading the data engineering team to deliver high-quality data solutions for our clients. Your responsibilities will include: - Leading and guiding the team to design, implement, and deliver complex cloud-based data engineering and data warehousing projects independently or with minimal support. - Expertise in conceptualizing solution architecture and low-level design using a variety of data engineering and cloud hosting technologies. - Managing projects within a fast-paced agile ecosystem to ensure timely delivery of quality solutions. - Responsible for risk management, maintaining risk documentation, and implemen...

As a skilled and experienced Technology Stack Administrator, your role will involve managing and optimizing the on-premises infrastructure comprising RedHat OpenShift, Apache Spark, Apache Airflow, Delta Lake, Apache Kafka, and Debezium. Your responsibilities will include: - Platform Administration: - Installing, configuring, and maintaining RedHat OpenShift clusters in an on-premises environment. - Building and orchestrating data pipelines using Apache Airflow. - Implementing and managing Delta Lake for ACID-compliant data lake architecture. - Managing Apache Spark clusters for distributed data processing. - Developing real-time data streaming and CDC solutions using Apache Kafka and Debezi...

You will be joining LSEG (London Stock Exchange Group), a global financial markets infrastructure and data business that is dedicated to delivering excellent services to its customers. With a presence across 70 countries, LSEG employs 25,000 people globally and values diversity and inclusion in its workforce. As a Software Data Engineer at FTSE Russell, you will be involved in designing and developing scalable data pipelines using Python and Apache Spark. Your responsibilities will include orchestrating data workflows with AWS tools like Glue, EMR Serverless, Lambda, and S3, applying modern software engineering practices, contributing to the development of a lakehouse architecture using Apac...

Job Title : Lead Data Engineer Python & GCP Experience : 1012 years Location : Hyderabad JD : We are looking for an experienced Lead Data Engineer with strong expertise in Python and Google Cloud Platform (GCP) . You will be responsible for designing, developing, and maintaining scalable ETL/ELT data pipelines and modern data architectures. This role involves end-to-end ownership — from requirement gathering to solution delivery and post-production support. Key Responsibilities: Design and build ETL/ELT pipelines using Python. Architect and implement scalable data solutions on GCP . Work with GCP services like: Dataflow (real-time/batch processing) BigQuery (data warehouse) Cloud Functions ,...

Job Title : Lead Data Engineer Python & GCP Experience : 10 to 15 years Location : Hyderabad Mode : Hybrid Notice Period : 30 days (serving Notice only ) About the Role: We are looking for a Lead Data Engineer who is great at Python and Google Cloud Platform (GCP) . You will design and build data pipelines , work with cloud services , and make sure data is clean, fast, and reliable . Responsibilities: Build and manage ETL/ELT pipelines using Python . Work with GCP tools like Dataflow, BigQuery, Cloud Functions, Composer, and GCS . Use Kafka, Spark, Airflow, Redis, MongoDB for data processing and orchestration. Deploy solutions using GKE or Cloud Run . Handle data ingestion, transformation, a...

Role & responsibilities Preferred candidate profile Job Title: Senior Data Engineer - Azure Stack We are looking for a highly skilled Senior Data Engineer with deep expertise in Azure Stack to join our growing team. As a Senior Data Engineer, you will be instrumental in designing and developing robust data pipelines, data lakes, and cloud-based solutions to support our business intelligence and analytics initiatives. You will work in a hybrid cloud environment to manage data services, ensuring high scalability, security, and performance for data-driven decision-making. Key Responsibilities: Data Engineering: Design and implement scalable data pipelines, ETL workflows, and processing framewor...

Project Duration 3 months and can get extended. Data Engineer (Databricks experience is mandatory) Exp: 3-8 Years Qualification: B. Tech/B.Sc. (IT) Expertise in Python, PySpark , and Apache Spark for data engineering Strong experience in building and optimizing ETL/data transformation pipelines Proficient in Databricks and Delta Lake storage format (Databricks experience is mandatory) Cloud platforms like AWS (Glue, EMR), Snowflake, Dataiku, Alteryx Skilled in managing large-scale, multi-dimensional datasets and designing scalable data architectures Solid understanding of data modeling and data warehouse concepts (On-Prem & Cloud) Familiar with modern data tools and continually updated on em...

As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable and efficient data pipelines using PySpark and Databricks. Your key responsibilities will include: - Performing data extraction, transformation, and loading (ETL) from diverse structured and unstructured data sources. - Developing and maintaining data models, data warehouses, and data marts in Databricks. - Being proficient in Python, Apache Spark, and PySpark. - Integrating third party data from multiple sources with internal data. - Writing and optimizing complex SQL queries for high performance and scalability across large datasets. - Collaborating closely with data scientists, analysts, and bu...

As an AWS Glue Developer with 6 to 9 years of experience, you will be responsible for the following: - Hands-on experience in AWS Glue ETL development for at least 3 years - Strong programming skills in Python and PySpark for data transformation - Proficiency in AWS services such as S3, Redshift, Athena, Lambda, and EMR - Experience working with data formats like Parquet, Avro, JSON, CSV, and ORC - Advanced SQL skills for data querying and transformation - Deep understanding of ETL/ELT design patterns and best practices - Knowledge of dimensional modeling, star/snowflake schemas - Experience with Git/Bitbucket for code management Preferred skills for this role include: - Experience with Vang...

Role Overview: You will be responsible for designing, developing, and maintaining data pipelines and solutions using Databricks, Apache Spark, and other related tools to process large datasets and extract actionable business insights. Your role will involve building metrics deck and dashboards for KPIs, including the underlying data models. Additionally, you will be required to design, implement, and maintain a platform providing secured access to large datasets. Key Responsibilities: - Hold a Bachelor's degree in computer science, Data Science, engineering, mathematics, information systems, or a related technical discipline - Possess 5+ years of relevant experience in data engineering roles...

As an AVP - Data Science at GenAI, you will be responsible for driving advanced analytics opportunities in the BFSI sector. Your role will involve leading end-to-end project delivery, stakeholder engagement, and team management to deliver high-impact solutions for Indian clients. **Key Responsibilities:** - Build strong partnerships with BFSI clients and channel partners - Communicate methodologies effectively to drive stakeholder confidence - Scope, plan, and deliver end-to-end analytics projects - Design and deploy high-impact data science and GenAI models - Mentor, coach, and upskill junior data scientists - Ensure adherence to SOPs, documentation, and governance frameworks **Qualificatio...

As an Engineering Manager (Data Engineering) at Amgen, your role will involve leading and mentoring a team of data engineers in the R&D domain of biotech or pharma companies. You will foster a culture of innovation, collaboration, and continuous learning to solve complex problems within the R&D division. Your responsibilities will include overseeing the development of data extraction, validation, and transformation techniques to ensure high-quality data for downstream systems. Additionally, you will guide the team in writing and validating high-quality code for data processing and transformation, as well as drive the development of data tools and frameworks for efficient data management acro...

Role Overview: As an AI Engineer at Merkle, your role involves leading and managing customer analysis deep dives, executing strategies outlined by senior leadership, and managing day-to-day communication with clients. Key Responsibilities: - Develop algorithms and build predictive models from the ground up, utilizing machine learning and deep learning frameworks - Extensive experience in building Information retrieval systems and relevance engineering - Building AI models from scratch and assisting product managers and stakeholders with analysis and implementation - Transform machine learning models into APIs for integration with other applications - Provide training for AI engineers on mode...

As a Big Data Engineer with 10+ years of experience, your role will involve handling Big Data projects using Azure technologies such as Azure Databricks, Azure Data Factory, and Azure Function apps. You will also work with Apache Spark, Scala, Java, Apache Kafka, and event stream technologies. Your key responsibilities will include: - Vent streaming using Kafka - Hands-on experience with Scala and Java programming, Spark framework, and performance optimization - Building reusable frameworks with high quality - Working on CI/CD using tools like Azure DevOps and GitHub, and following best practices, code quality, and branching strategies - Automating tasks and deploying production standard cod...

Role Overview: As a Data Intelligence & AI Solution Architect at Capgemini, you will lead the design and implementation of advanced data platforms and AI-driven solutions. You will collaborate with presales, delivery, and client teams to architect scalable systems using Databricks, MLOps frameworks, and cloud-native technologies. Key Responsibilities: - Architect and design data intelligence platforms using Databricks, Apache Spark, and Delta Lake. - Lead MLOps initiatives to streamline model development, deployment, and monitoring. - Collaborate with account teams to understand client challenges and propose AI-driven solutions. - Support presales activities including RFP responses, solution...

You will be responsible for designing, building, and deploying data pipelines and workflows using Databricks. Additionally, you will develop and optimize Apache Spark applications for data processing and analytics. You will integrate Databricks with various data sources and platforms, implement data quality checks and monitoring processes, and manage/configure the Databricks environment including clusters and notebooks. Troubleshooting and resolving issues related to Databricks performance and functionality will also be part of your role. Furthermore, collaborating with data scientists and analysts to implement machine learning models and staying up-to-date with the latest Databricks feature...

As a Senior Data Engineer (Scala) at our technology services & software exports company, your primary responsibility will be to design, develop, and maintain scalable data pipelines using Scala and Apache Spark for batch and streaming workloads. Your role also involves implementing Kafka-based ingestion, processing streams with structured streaming, and ensuring data reliability across the analytics lifecycle. You will be expected to author efficient Spark SQL transformations, optimize job performance, and resource utilization on cluster platforms. Additionally, building robust ETL workflows, handling schema evolution, and implementing data quality checks and monitoring will be part of your ...

As an Azure Databricks+ MS Fabrics+ Python Data Engineer at PWC AC, you will play a crucial role in designing and implementing scalable ETL/ELT pipelines using Databricks and Apache Spark. Your responsibilities will include: - Proficiency with Azure Data Services, Databricks, and Snowflake - Deep understanding of traditional and modern data architecture and processing concepts - Hands-on experience with Snowflake utilities, SnowSQL, SnowPipe, ETL data pipelines, and Big Data model techniques using Python / Java - Building stream-processing systems using solutions such as Storm or Spark-Streaming - Experience with Big Data ML toolkits like Mahout, SparkML, or H2O - Knowledge in Python and cer...

As a Senior Lead Machine Learning Engineer at our client, a Global leader in financial intelligence, data analytics, and AI-driven solutions, you will be at the forefront of building cutting-edge ML-powered products and capabilities for natural language understanding, information retrieval, and data sourcing solutions. Your role in the Document Platforms and AI team will involve spearheading the development and deployment of production-ready AI products and pipelines, mentoring a talented team, and playing a critical role in shaping the future of global markets. **Responsibilities:** - Build production ready data acquisition and transformation pipelines from ideation to deployment - Be a han...

In this role, you will ensure effective Design Development Validation and Support activities to meet client needs and architectural requirements. Your responsibilities will include gathering requirements, translating them into system requirements, and contributing to project estimations. Additionally, you will play a key role in building efficient programs and systems to support clients in their digital transformation journey. **Key Responsibilities:** - Ensure effective Design Development Validation and Support activities - Gather and translate client requirements into system requirements - Contribute to project estimations - Build efficient programs and systems to support clients" digital ...

As a Senior Spark Data Engineer at Barclays, you will play a crucial role in the evolution of the digital landscape, driving innovation and excellence. Your main responsibilities will include: - Designing, building, and maintaining data processing systems using Apache Spark and related big data technologies. - Ensuring data quality, integrity, and security throughout the data lifecycle. - Developing and maintaining data processing frameworks and reusable components. - Creating and maintaining documentation for data pipelines and processes. - Supporting the migration of legacy data processing systems to modern big data platforms. - Utilizing technologies such as Spark, Hive, Solace, Kafka, Ja...

As a Data Analyst at Allime Tech Solutions, your role will involve working with data processing and analysis using Data Bricks in an Azure environment. Your responsibilities will include: - Developing data pipelines and managing data infrastructure within Azure - Implementing and optimizing data solutions using multiple programming languages - Working on end-to-end Data Product deployment - Collaborating with cross-functional teams for reporting, analysis, and MIS projects To excel in this role, you must have: - A Bachelor's or Master's degree in Computer Science, Information Technology, or a related field - 7+ years of experience as a Data Analyst, with 3 years specifically in banking data ...

Role Overview: As a Staff Backend Engineer at Celonis, you will utilize your general software engineering expertise to contribute to innovative projects and evolving concepts. You will have the opportunity to work with cutting-edge cloud technology, stay updated on trends, and drive projects with increasing independence. This role offers a unique chance for talented engineers to expand their boundaries and think innovatively. Key Responsibilities: - Innovate beyond existing products and bring new ideas to life for Celonis. - Take ownership of implementation and delivery of assigned projects. - Demonstrate technical leadership within the engineering team. - Ensure high quality and continuousl...

As a Pyspark Developer at our company, you will be responsible for optimizing Pyspark applications to improve performance and resource utilization, ensuring efficient data processing and analysis. Your key responsibilities will include: - Developing and maintaining scalable solutions using AWS EC2 and AWS EMR to support business needs and data-driven decision-making. - Implementing PySpark frameworks to process large datasets, enhancing data accessibility and insights for stakeholders. - Collaborating with cross-functional teams to design and deploy robust data pipelines, ensuring seamless integration with existing systems. - Utilizing Python programming to automate tasks and streamline work...