905 Apache Spark Jobs - Page 10

Role Overview: As a member of our team at Quantiphi, you will be responsible for creating and maintaining LookML code to define data models, dimensions, measures, and relationships within Looker. Your role will involve developing reusable LookML components to ensure efficiency in report and dashboard creation, building and customizing dashboards with data visualizations, writing complex SQL queries, optimizing Looker performance, configuring user roles and permissions, and providing training and support to business users. Key Responsibilities: - Create and maintain LookML code to define data models, dimensions, measures, and relationships within Looker. - Develop reusable LookML components f...

As a Lead Data AI/ML Engineer - Vice President at Barclays, you will spearhead the evolution of the Corporate Data Services function. Your role will involve effective stakeholder management, leadership, and decision-making to support business strategy and risk management. **Key Responsibilities:** - Participating in daily stand-up meetings to discuss progress, blockers, and plans for the day. - Working on deploying AI/ML models into production environments using AWS services like Glue, Lambda, and Step Functions. - Building and maintaining data pipelines using Spark and Python to ensure seamless data flow and integration. - Collaborating with data scientists, software engineers, and other st...

Role Overview: The Distributed Data Systems Team at Dataflare is seeking talented engineers who are passionate about designing and deploying distributed data systems at PB cloud-scale. As a Team Lead, you will play a crucial role in understanding business requirements, mentoring a team of engineers, designing high-performance storage algorithms, and contributing to the foundational programming model of the data platform. Key Responsibilities: - Understand and craft solutions based on business requirements - Mentor and develop a team of engineers - Design and implement performant, parallelized, and multi-threaded storage algorithms - Contribute to the scheduling and programming model for all ...

As a Data Services Engineering Lead joining our innovative development team, you will have the opportunity to lead the design and implementation of cutting-edge data solutions that drive business intelligence and analytics. Your responsibilities will include: - Leading the design, development, and deployment of scalable Java applications integrated with Databricks for big data processing - Architecting and optimizing distributed computing solutions using Apache Spark within the Databricks environment to handle large-scalable datasets efficiently - Implementing and managing data lake house architecture using Databricks, ensuring data integrity, security, and accessibility - Developing and mai...

As a Senior Data Engineer at Enable Data Incorporated, you will be a key player in designing, developing, and maintaining scalable and efficient data solutions in the cloud. Leveraging your expertise in cloud technologies, Spark, and Databricks, you will collaborate with cross-functional teams to gather data requirements, design and implement data processing pipelines, and ensure data quality and integrity. Your role will involve optimizing data infrastructure for clients and providing advanced analytics insights. If you are passionate about using cutting-edge technologies to solve complex data challenges, this is the perfect opportunity for you. **Key Responsibilities:** - Design, develop, ...

Role & responsibilities Around 6+ years of experience in data engineering or cloud computing data development Design, develop, and optimize ETL/ELT data pipelines using Apache Spark (PySpark or Scala), AWS Glue, and Azure Data Factory Work with structured and unstructured data to build scalable ingestion and transformation workflows across cloud platforms. Build data lake and data warehouse solutions using AWS S3, Azure Data Lake Collaborate with data scientists, analysts, and application developers to support advanced analytics, reporting, and ML workflows Implement job orchestration, monitoring, and error handling for reliable pipeline execution. Maintenance and support of Data Pipeline. D...

Experience in developing and delivering scalable big data pipelines using Apache Spark and Databricks on AWS. Position Requirements : Must Have : Build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimize ETL/ELT processes for structured and unstructured data. Knowledge of Lakehouse architecture for efficient data storage, processing, and analytics. Orchestrating ETL/ELT Pipelines: Design and manage data workflows using Databricks Workflows, Jobs API. Worked with AWS Data Services (S3, Lambda, CloudWatch) for seamless integration. Performance Optimization: Optimize queries using pushdown capabilities and indexing strategies. Worked on data governance w...

Role Overview: You will be responsible for integrating data from various siloed systems into CDF using ETL tools such as Azure Data Factory or Informatica PowerCenter. Additionally, you will transform and contextualize data within CDF using programming languages like Python and tools like Apache Spark and Databricks for data analysis and visualization. Your understanding of industrial data and processes will be crucial for successful data integration. Key Responsibilities: - Utilize ETL tools like Azure Data Factory or Informatica PowerCenter to integrate data from different systems into CDF - Write scripts in Python, especially for using the Cognite Python SDK, to transform and contextualiz...

As an experienced Azure Data Engineer with over 8 years of experience, you will play a key role in designing and implementing scalable data pipelines and ETL processes within the Azure ecosystem. Your expertise in data warehousing, data pipelines, and cloud-based data processing frameworks will be crucial for this role. Key Responsibilities: - Design and implement scalable data pipelines and ETL processes using Azure Data Factory (ADF), PySpark, and Databricks. - Manage and optimize Azure Data Lake and integrate with Azure Synapse Analytics for large-scale data storage and analytics. - Collaborate with cross-functional teams to gather requirements, design data solutions, and deliver actionab...

As a Senior Data Engineer, you will be responsible for: - Designing and implementing solutions for problems arising out of large-scale data processing - Attending/driving various architectural, design, and status calls with multiple stakeholders - Ensuring end-to-end ownership of all tasks aligned, including development, testing, deployment, and support - Designing, building, and maintaining efficient, reusable, and reliable code - Testing implementation, troubleshooting, and correcting problems - Working both as an individual contributor and within a team - Ensuring high-quality software development with complete documentation and traceability - Fulfilling organizational responsibilities by...

As a Data Engineer with 4-5 years of experience, you will be responsible for designing and developing scalable data infrastructure and systems to support critical business functions. Your role will involve building robust ETL pipelines, managing large-scale datasets, and optimizing data workflows in a fast-paced environment. Key Responsibilities: - Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data. - Model and structure data for performance, scalability, and usability. - Work with cloud infrastructure, preferably Azure, to optimize data workflows. - Build and manage data lake/lakehouse architectures following best practices. - Optimize...

Role Overview: As a software engineer in Corporate Planning and Management (CPM) Engineering, you will be involved in creating new financial and spend management workflows and developing intelligent reporting systems to drive commercial benefits for the firm. You will have the opportunity to work in small, agile teams and contribute to various aspects of corporate planning and management in alignment with the firm's strategic objectives. Key Responsibilities: - Be energetic, self-directed, and self-motivated, capable of building and maintaining long-term relationships with clients and colleagues - Approach problems with an open mind, working collaboratively within a team - Utilize exceptiona...

As a member of the Data Core Apps Team at Agoda, you will be part of a skilled, driven, and diverse team dedicated to making an impact in the travel industry. You will have the opportunity to work on cutting-edge technology and contribute to the development of first-class big data tools. Here is what you can expect in this role: Role Overview: - Work with a technology stack including Scala, Spark, Python, Hadoop, Kubeflow, SQL, S3 - Enhance scalability, stability, accuracy, speed, and efficiency of existing data systems - Design, build, test, and deploy new libraries, frameworks, or full systems while maintaining high standards of testing and code quality - Collaborate with experienced engin...

As a Machine Learning Engineer at Tecblic, you will be part of a forward-thinking team that specializes in delivering AI-driven solutions to empower businesses in the digital age. Your primary responsibilities will include: - Research and Development: Design, develop, and fine-tune machine learning models across LLM, computer vision, and Agentic AI use cases. - Model Optimization: Fine-tune and optimize pre-trained models for improved performance, scalability, and minimal latency. - Computer Vision: Build and deploy vision models for tasks such as object detection, classification, OCR, and segmentation. - Integration: Collaborate closely with software and product teams to integrate models in...

As a Machine Learning Scientist I at Wayfair in Bangalore, India, your role will involve designing, building, deploying, and refining large-scale machine learning/AI models and algorithmic decision-making systems to solve real-world problems for customers. You will work cross-functionally with commercial stakeholders, engineers, analysts, and product managers to develop ML solutions that drive real-world impact. Collaboration with various infrastructure and ML platform teams is essential to develop production-grade services following best practices. Additionally, you will be responsible for building cutting-edge robust ML models that directly impact customers every day and identifying new op...

You will capture user requirements and translate them into business and digitally enabled solutions across a range of industries. Your responsibilities will include: - Designing, developing, optimizing, and maintaining data pipelines that adhere to ETL principles and business goals - Solving complex data problems to deliver insights that help our business achieve its goals - Sourcing data (structured and unstructured) from various touchpoints, formatting, and organizing them into an analyzable format - Creating data products for analytics team members to improve productivity - Calling AI services like vision, translation, etc., to generate an outcome that can be used in further steps along t...

As a Software Engineering Manager at SAP, you will lead the App2App Integration team within the SAP Business Data Cloud. Your role will involve driving the design, development, and delivery of integration services that facilitate real-time, secure, and scalable data exchange across SAP's enterprise applications and unified data fabric. **Key Responsibilities:** - Own end-to-end delivery of App2App integration components and services using Java, Apache Kafka, RESTful APIs, and modern data pipelines. - Accelerate the App2App integration roadmap by identifying reusable patterns, driving platform automation, and establishing best practices. - Drive the evolution of ETL pipelines and real-time da...

As a Technology Stack Administrator at Bajaj Allianz Life Insurance, your role involves managing and optimizing the on-premises infrastructure comprising RedHat OpenShift, Apache Spark, Apache Airflow, Delta Lake, Apache Kafka, and Debezium. Your responsibilities include: - **Platform Administration:** - Install, configure, and maintain RedHat OpenShift clusters in an on-premises environment. - Build and orchestrate data pipelines using Apache Airflow. - Implement and manage Delta Lake for ACID-compliant data lake architecture. - Manage Apache Spark clusters for distributed data processing. - Administer and maintain Apache Spark clusters, ensuring optimal performance and resource utilization...

Role Overview: You will be part of a dynamic data engineering team as a skilled Data Engineer, focusing on designing, developing, and maintaining scalable data pipelines and data lake solutions. Your expertise in Azure Data Services, Apache Spark, and Big Data technologies will be key in ensuring the availability, quality, and reliability of data for analytics and AI workloads. Collaboration with data scientists, analysts, and business stakeholders will be essential to drive successful outcomes. Key Responsibilities: - Design, build, and maintain scalable and reliable data pipelines utilizing Azure Data Factory, Azure Synapse, and Databricks. - Develop and optimize large-scale data processin...

As a Senior Engineer at Impetus Technologies, you will be responsible for designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your role will involve collaborating with cross-functional teams to gather requirements, develop high-quality code, and optimize data processing workflows. Additionally, you will mentor junior engineers and contribute to architectural decisions to enhance system performance and scalability. Key Responsibilities: - Design, develop, and maintain high-performance applications using Java and Big Data technologies. - Implement data ingestion and processing workflows utilizing frameworks like Hadoop and Spark. - C...

As an Engineer with 5-8 years of experience, your role will involve working with competencies like Apache Beam, GCP Cloud, and Open Shift. You should have hands-on experience in Java and Kafka Streaming, with knowledge in GCP and Kubernetes being a plus. Your responsibilities will include developing and deploying data and analytics-led solutions on GCP, as well as designing highly available and scalable systems. You will be working on Data Engineering solutions using Cloud BigQuery, Cloud DataFlow, Cloud BigTable, Storage, Cloud Spanner, and Cloud IAM. It's essential to have a good understanding of Apache Kafka and proficiency in cloud-based ETL/Data Orchestration tools like Apache Beam and ...

You are invited to join a dynamic team for a long-term project as a Spark & Databricks Developer. Your role involves designing, developing, and optimizing big data pipelines using Apache Spark, building scalable data solutions on Databricks, collaborating with cross-functional teams for data integration, and managing version control using GitHub. Your responsibilities also include ensuring data quality, performance tuning, and participating in code reviews, testing, and documentation. Key Responsibilities: - Design, develop, and optimize big data pipelines using Apache Spark - Build and maintain scalable data solutions on Databricks - Collaborate with cross-functional teams for data integrat...

Job Title: Spark Data Engineer SME (Part-Time) Remote (India) | Work From Home | Flexible Timings About the Opportunity We are looking for experienced Spark Data Engineers (5+ years) to create content and mentor future data professionals. This role is ideal for professionals with strong expertise in Apache Spark and big data engineering who want to share knowledge and guide learners. What Were Looking For 5+ years of hands-on experience in data engineering Strong expertise in Apache Spark (PySpark/Scala/Java) Experience with ETL workflows, big data platforms, and cloud deployments Solid SQL and data modeling skills Interest in mentoring and simplifying complex concepts Why Join Us Part-time ...

Position Summary We are looking for a highly skilled and experienced Data Engineering Manager to lead our data engineering team. The ideal candidate will possess a strong technical background, strong project management abilities, and excellent client handling/stakeholder management skills. This role requires a strategic thinker who can drive the design, development and implementation of data solutions that meet our clients needs while ensuring the highest standards of quality and efficiency. Job Responsibilities Technology Leadership Lead guide the team independently or with little support to design, implement deliver complex cloud-based data engineering / data warehousing project assignment...

We are seeking an experienced Java Technical Lead with 8-12 years of experience for an immediate joiner position in Noida. The role involves leading a technical team in the design, development, and implementation of scalable Java-based enterprise solutions using Agile methodologies. Responsibilities include defining architecture, capturing requirements, leveraging Spring Boot and Spring Cloud frameworks, performing code reviews, implementing PoCs, troubleshooting bottlenecks, and ensuring system scalability and performance. The candidate should possess strong expertise in distributed architecture, multithreading, RDBMS, and NoSQL databases, with added knowledge in Apache Spark, Data Science,...