876 Apache Spark Jobs - Page 8

Name of Position: Databricks Lead No. of Positions : 1 Company Name: Whiteblue Cloud Services Location: Chennai, Tamil Nadu Job Description Role Overview: We are looking for an experienced Databricks Lead to design, develop, and optimize large-scale data solutions leveraging Azure Databricks, Azure Data Factory, and PySpark . The ideal candidate will have strong leadership skills, hands-on technical expertise, and a passion for driving cloud-based data modernization initiatives. Key Responsibilities Lead end-to-end data engineering and modernization projects using Azure Databricks and related technologies. Design, develop, and maintain scalable data pipelines and workflows using Azure Data F...

Description The Data Engineering Lead is a senior-level position within our Data Engineering team, responsible for spearheading the design, implementation, and optimization of advanced data architectures using Databricks. This role involves leading a team of data engineers, collaborating with business and data analysts, and business stakeholders, and delivering scalable, secure data solutions tailored to the needs of commercial analytics in the pharmaceutical industry. Responsibilities 1. Architecture Design Develop and maintain scalable data architectures using Databricks, including data lakes, data warehouses, and real-time processing systems. Create detailed blueprints for data processes ...

Role: Data Engineering Lead Location: Bangalore (Work from Office) Experience: 12+ Years Notice Period: Immediate to 30 Days Job Description: Design, develop, and optimize scalable data architectures using Databricks and Apache Spark . Build and maintain ETL/ELT pipelines for large-scale data processing. Ensure data governance , security , and compliance across the Databricks environment. Collaborate with analytics, marketing, and business teams to deliver data-driven solutions. Integrate Databricks with AWS/Azure/GCP , and CRM tools like Salesforce or Veeva . Lead and mentor a team of data engineers, ensuring delivery excellence. Implement Medallion Architecture and Delta Lake for efficient...

In this role, you will be developing and implementing quantitative software applications to process and analyze real-time financial market data in a high-performance computing environment. Your responsibilities will include maintaining and optimizing existing software applications, as well as recommending and implementing improvements. You will design and develop solutions using Python and PySpark in a spark/Hadoop environment. Additionally, you will build, optimize, and troubleshoot ETL workflows to extract, transform, and load data from diverse sources into big data platforms. Collaborating with data engineers, data scientists, and business stakeholders to gather requirements and deliver d...

As a Data Engineer, your role involves building products in the data analytics space with a focus on shipping high-quality data products and contributing to the overall company and product development. You should possess the right attitude, aptitude, skills, empathy, compassion, and a hunger for learning. Here are the key responsibilities and qualifications required for this position: **Key Responsibilities:** - Understand business requirements and implement scalable solutions. - Translate complex technical and functional requirements into detailed designs. - Develop a highly scalable, reliable, and high-performance data processing pipeline for extracting, transforming, and loading data from...

Role Overview: As an Applications Development Group Manager at Citigroup, you will be responsible for managing a team to establish and implement new or revised application systems and programs in coordination with the Technology Team. Your role involves managing multiple teams to achieve established goals, conducting personnel duties such as performance evaluations, hiring, and disciplinary actions, and providing strategic influence to exercise control over resources and budget management while monitoring end results. You will utilize your in-depth knowledge to resolve issues, contribute to defining standards, and integrate applications development knowledge with the overall technology funct...

As a Data Science/AI/ML professional at Cybage in Pune, you will have the opportunity to work with a leading company specialized in outsourced product engineering services. Cybage collaborates with top independent software vendors and utilizes a data-driven system called Decision Mines for Digital Excellence to provide innovative solutions leveraging modern technologies. Key Responsibilities: - Design, implement, and optimize ML pipelines with tools like TensorFlow, PyTorch, MLflow, Scikit-learn, or ONNX. - Operationalize models using CI/CD for ML practices (MLOps) on platforms like Kubeflow, Azure ML, or SageMaker. - Collaborate with data scientists to convert prototypes into production-lev...

You are an accomplished and dynamic Data Engineering Lead responsible for driving the design, development, and delivery of cutting-edge data solutions. Your role involves overseeing the design and development of robust data pipelines and systems, managing and mentoring a team of 5 to 7 engineers, and playing a critical role in architecting innovative solutions tailored to client needs. You will lead by example, fostering a culture of accountability, ownership, and continuous improvement while delivering impactful, scalable data solutions in a fast-paced consulting environment. Key Responsibilities: - Architecture & Delivery: - Lead the end-to-end architecture of data pipelines leveraging Azu...

As an AI Architect specializing in Agentic AI and Generative AI, your primary responsibility will be to design, develop, and deploy cutting-edge autonomous AI systems. You will focus on building LLM-powered agents with memory, tool usage, planning, and reasoning capabilities to create intelligent, goal-oriented systems. Your role as a technical leader will involve overseeing AI initiatives from research and architecture design to deployment in cloud environments. Key Responsibilities: - Architect and construct LLM-based agents capable of autonomous task execution, memory management, tool usage, and multi-step reasoning. - Develop modular, goal-oriented agentic systems utilizing tools like La...

As a member of the Managed Services team at PwC, you will focus on providing outsourced solutions and supporting clients across various functions. Your role will involve streamlining operations, reducing costs, and enhancing efficiency by managing key processes and functions on behalf of organizations. You will leverage your skills in project management, technology, and process optimization to deliver high-quality services to clients. Specifically, in managed service management and strategy, your responsibilities will include transitioning and running services, managing delivery teams, programmes, commercials, performance, and delivery risk. Your work will revolve around continuous improveme...

As a Data Engineer II, your role involves managing the deprecation of migrated workflows and ensuring the seamless migration of workflows into new systems. Your expertise in building and maintaining scalable data pipelines, both on-premises and on the cloud, will be crucial. You should have a deep understanding of input and output data sources, upstream downstream dependencies, and data quality assurance. Proficiency in tools like Git, Apache Airflow, Apache Spark, SQL, data migration, and data validation is essential for this role. Your key responsibilities will include: - Workflow Deprecation: - Evaluate current workflows" dependencies and consumption for deprecation. - Identify, mark, and...

As a Senior Data Engineer specializing in Data Quality, Ingestion & API Development, your role will involve leading the development of a scalable data ingestion framework with a strong focus on ensuring high data quality and validation. You will also be responsible for designing and implementing robust APIs for seamless data integration. Your expertise in building and managing big data pipelines using modern AWS-based technologies will be crucial for driving quality and efficiency in data processing systems. - **Data Ingestion Framework:** - **Design & Development:** Architect, develop, and maintain an end-to-end data ingestion framework that efficiently extracts, transforms, and loads data ...

As a senior member of the team responsible for developing large-scale, highly available, and fault-tolerant next-generation SaaS solutions for Health Care Analytics, your role will involve the following key responsibilities: - Design and implement scalable, secure, and maintainable enterprise-level solutions. - Establish coding standards, best practices, and architectural guidelines. - Integrate business process management solutions with existing systems. - Ensure system architectures align with business requirements and technology roadmap. Your core technical skills should include: - Advanced Java development with extensive experience in Java 21 features. - Deep understanding of Spring Fram...

About the role: Data Management roles are skills and expertise in the secure storage and management of data in compliance with guidelines, standards and best practice They enable efficient and high quality data analysis by designing and implementing approaches to clean and transform data and store data in the most appropriate way Purpose of Position: We are seeking a highly skilled and experienced Data Engineering Lead to spearhead our data architecture strategy, oversee robust data engineering initiatives, and manage data lake operations The ideal candidate will be a hands-on leader with deep expertise in building and optimizing data platforms that support advanced analytics and business in...

Role Overview: At PwC, your focus in managed services will involve providing outsourced solutions and supporting clients across various functions. You will assist organisations in streamlining operations, reducing costs, and enhancing efficiency by managing key processes and functions on their behalf. Your expertise in project management, technology, and process optimization will enable you to deliver high-quality services to clients. As part of the managed service management and strategy team at PwC, you will be responsible for transitioning and running services, managing delivery teams, programmes, commercials, performance, and delivery risk. Your role will revolve around continuous improv...

Role Overview: As a Spark (Scala) Developer at Viraaj HR Solutions, your primary responsibility is to design, develop, and implement data processing applications using Apache Spark and Scala. You will work closely with cross-functional teams to gather requirements, define project specifications, and build scalable ETL pipelines for efficient data processing. Your role also involves optimizing Spark jobs for performance, utilizing SQL for data querying and transformation, and developing data models to support analytics initiatives. Additionally, you will be engaged in data analysis, integration of big data tools and frameworks, code reviews, troubleshooting application issues, and ensuring hi...

You will be joining GlobalLogic to work on a significant software project for a world-class company specializing in M2M / IoT 4G/5G modules for industries like automotive, healthcare, and logistics. Your role will involve contributing to the development of end-user modules" firmware, implementing new features, ensuring compatibility with the latest telecommunication and industry standards, and analyzing customer requirements to provide estimations. - Design end-to-end AI/ML architecture and solutions in alignment with business objectives - Lead the development and deployment of ML models using frameworks like TensorFlow, PyTorch, or Scikit-learn - Architect scalable data pipelines for traini...

Role Overview: Do you aspire to drive the next level of application architecture in the Compliance IT space Are you passionate about applying Agile methodologies at scale and developing a cutting-edge Data Ingestion, Aggregation & Calculation Platform on Cloud Would you like to contribute to shaping the future of Compliance technology alongside a dynamic cross-functional team If so, we have an exciting opportunity for you as a seasoned full stack developer. In this role, you will be responsible for implementing and delivering high-quality software solutions for our state-of-the-art Data Ingestion, Aggregation & Calculation platform. Key Responsibilities: - Implement and deliver high-quality ...

**Role Overview:** As a Data Engineer, your primary responsibility will involve designing, developing, and maintaining ETL processes to facilitate the smooth integration of raw data from diverse sources into data lakes or warehouses. You will be leveraging tools such as Python, PySpark, SQL, and various AWS services like Lambda, Glue, Redshift, and S3 to efficiently process, analyze, and store large-scale datasets. Additionally, you will be tasked with developing and optimizing data pipelines using AWS Glue, PySpark, and Python, while also utilizing your expertise in Apache Spark/Databricks for advanced ETL workflows. Writing and managing SQL queries for data retrieval, transformation, and s...

Job Description: As a Principal Software Engineer, you will provide technical leadership within development teams, ensuring architectural coherence, collaboration, and best practices. Your role involves bridging the gap between data engineering, data science, and software engineering to help build scalable, maintainable data solutions. You will design and implement solutions, mentor developers, influence technical direction, and drive best practices. Additionally, you will have line management responsibilities, focusing on the growth and development of team members. The role will involve working in an Azure and Databricks environment, leveraging cloud-native technologies and modern data plat...

As the Director of Data and Analytics at KASH Tech, your role will involve driving the strategy, execution, and delivery of data and analytics solutions across modern cloud platforms, AI/ML, and big data ecosystems. Your responsibilities will include defining and executing data modernization strategies, leading enterprise cloud data transformations, developing best-fit methodologies for client solutions, identifying opportunities for AI, automation, and data-driven decision-making, and architecting and implementing scalable data platforms using various tools like Databricks, Azure Synapse, and Microsoft Fabric. Key Responsibilities: - Define and execute data modernization strategies aligned ...

Role Overview: As a member of our team at Quantiphi, you will be responsible for creating and maintaining LookML code to define data models, dimensions, measures, and relationships within Looker. Your role will involve developing reusable LookML components to ensure efficiency in report and dashboard creation, building and customizing dashboards with data visualizations, writing complex SQL queries, optimizing Looker performance, configuring user roles and permissions, and providing training and support to business users. Key Responsibilities: - Create and maintain LookML code to define data models, dimensions, measures, and relationships within Looker. - Develop reusable LookML components f...

As a Lead Data AI/ML Engineer - Vice President at Barclays, you will spearhead the evolution of the Corporate Data Services function. Your role will involve effective stakeholder management, leadership, and decision-making to support business strategy and risk management. **Key Responsibilities:** - Participating in daily stand-up meetings to discuss progress, blockers, and plans for the day. - Working on deploying AI/ML models into production environments using AWS services like Glue, Lambda, and Step Functions. - Building and maintaining data pipelines using Spark and Python to ensure seamless data flow and integration. - Collaborating with data scientists, software engineers, and other st...

Role Overview: The Distributed Data Systems Team at Dataflare is seeking talented engineers who are passionate about designing and deploying distributed data systems at PB cloud-scale. As a Team Lead, you will play a crucial role in understanding business requirements, mentoring a team of engineers, designing high-performance storage algorithms, and contributing to the foundational programming model of the data platform. Key Responsibilities: - Understand and craft solutions based on business requirements - Mentor and develop a team of engineers - Design and implement performant, parallelized, and multi-threaded storage algorithms - Contribute to the scheduling and programming model for all ...

As a Data Services Engineering Lead joining our innovative development team, you will have the opportunity to lead the design and implementation of cutting-edge data solutions that drive business intelligence and analytics. Your responsibilities will include: - Leading the design, development, and deployment of scalable Java applications integrated with Databricks for big data processing - Architecting and optimizing distributed computing solutions using Apache Spark within the Databricks environment to handle large-scalable datasets efficiently - Implementing and managing data lake house architecture using Databricks, ensuring data integrity, security, and accessibility - Developing and mai...