905 Apache Spark Jobs - Page 15

As a Databricks Architect at NTT DATA, your role involves leading the design and implementation of scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform. You will define architectural standards, guide engineering teams, and collaborate with stakeholders to align data strategies with business goals. Your main focus will be leveraging Databricks unified analytics capabilities to build enterprise-grade data platforms supporting advanced analytics, machine learning, and real-time data processing. Key Responsibilities: - Define and own the end-to-end architecture of data platforms on Databricks, covering ingestion, transformation, storage, and consumption l...

Role Overview: As an AVP, Business Intelligence Developer at Synchrony, you will be responsible for designing, developing, and implementing Natural Language Processing (NLP) solutions to enhance efficiency and profitability across Operations. You will work with internal stakeholders to acquire and refine business requirements for business intelligence solutions in the financial industry. Your role will involve working in the Apache Spark framework and its machine learning library (Spark ML) to build scalable NLP pipelines using Python and PySpark. Key Responsibilities: - Identify insights and opportunities for increased efficiency and growth across functions in operations. - Build and analyz...

You will be working as a Databricks Developer with 3-6 years of experience, located in India. Joining the data engineering and AI innovation team, your main responsibilities will include: - Developing scalable data pipelines using Databricks and Apache Spark - Implementing AI/ML workflows with tools like MLflow and AutoML - Collaborating with data scientists to deploy models into production - Performing ETL development, data transformation, and model training pipelines - Managing Delta Lake architecture - Working closely with cross-functional teams to ensure data quality and governance Qualification Required: - Bachelor's degree in Computer Science or a related field - Proven experience work...

Role Overview: You will be leading the development of scalable, secure, and high-performance data solutions as a Lead Data Engineer. Your role involves utilizing your technical expertise in Python, Apache Spark, Delta Lake, and orchestration tools like Databricks Workflows or Azure Data Factory to ensure data governance, metadata management, and regulatory compliance within insurance and financial services sectors. Key Responsibilities: - Design and architect end-to-end data engineering solutions across cloud platforms. - Create and manage robust data pipelines and ETL workflows using Python and Apache Spark. - Implement scalable Delta Lake solutions for structured and semi-structured data. ...

Job Summary: We are looking for an experienced Data Engineer who is proficient in designing and building data pipelines, data platforms, and data analysis solutions on the AWS ecosystem. The successful candidate will be the single point of contact for all data-related queries and pipeline delivery for the application products. Key Responsibilities: Design, build and maintain data pipelines and data platforms on AWS components, including AWS Data Lake, AWS Athena, Kafka, AWS Kinesis, AWS Aurora, AWS Document DB, Apache NiFi, Apache Spark, Apache Zookeeper, AWS S3, AWS EMR, and AWS Glue. Act as the single source of truth for all DBAT product data and handle both structured and unstructured dat...

Hiring: Data Engineers (30 Openings | Immediate Joiners Only) Data Engineers across large-scale systems. The Data Engineer is responsible for designing, building, and maintaining the data infrastructure and pipelines that enable efficient collection, storage, and analysis of large datasets. This role works closely with data scientists, analysts, and business teams to ensure data availability, quality, and scalability to support data-driven decision-making. Key Responsibilities: Design, develop, and maintain scalable data pipelines for ingesting and processing structured and unstructured data. Build and optimize data architectures, including databases, data lakes, and warehouses. Ensure data ...

The Role: Company Overview : Kennect is the Modern Sales Compensation Platform designed for enterprises. We are leading the way in Sales Performance Management, with everything businesses need to harness the power of Sales Incentives. Our mission is to deliver customised enterprise-level commission calculation and tracking software for the most innovative businesses around the world. Key Responsibilities: Writing well-designed, testable and efficient code. Building reusable components and libraries for future. Troubleshooting and debugging to optimize performance Providing code documentation and other inputs to technical documents and participating in code reviews. Working on big projects wi...

Role Overview: As a backend and data engineer at Agoda, you will play a vital role in developing robust backend systems, architecting scalable data pipelines, and maintaining storage solutions. Your expertise will be crucial in optimizing financial technology solutions and driving informed business decisions through reliable data insights. Key Responsibilities: - Own the full life cycle of products, from business requirements to proper monitoring - Design, develop, and maintain platforms and data pipelines across fintech - Build stable, scalable, and highly performant systems to meet dynamic demands - Mentor team members and collaborate with other teams and departments - Foster cross-functio...

Role Overview: At CGI, we are seeking a skilled and proactive Senior Software Engineer - Data Engineer with experience in Python, Kafka Streaming, and Spark. As part of the team, you will be responsible for designing, developing, and deploying scalable backend systems and real-time data pipelines in cloud-native environments. This role is ideal for individuals passionate about building high-performance systems with cutting-edge technologies. Key Responsibilities: - Develop real-time data streaming applications using Apache Kafka and Kafka Streams. - Build and optimize large-scale batch and stream processing pipelines with Apache Spark. - Containerize applications and manage deployments using...

As a Big Data Engineer, you will play a crucial role in developing and managing our company's Big Data solutions. Your responsibilities will include: - Meeting with managers to determine the company's Big Data needs. - Developing Big Data solutions on AWS, utilizing tools such as Apache Spark, Databricks, Delta Tables, EMR, Athena, Glue, and Hadoop. - Loading disparate data sets and performing pre-processing services using Athena, Glue, and Spark. - Collaborating with software research and development teams. - Building cloud platforms for company application development. - Maintaining production systems. To excel in this role, you are required to meet the following qualifications: - 5+ years...

As a Scala Developer at our company, you will play a crucial role in designing, building, and enhancing our clients" online platform to ensure optimal performance and reliability. Your responsibilities will include researching, proposing, and implementing cutting-edge technology solutions while adhering to industry best practices and standards. You will be accountable for the resilience and availability of various products and will collaborate closely with a diverse team to achieve collective goals. - Over 7 years of experience in crafting scalable and high-performance backend systems - Expertise in functional programming and familiarity with contemporary data processing frameworks - Profici...

As a Lead Data Engineer, you will be responsible for designing and developing end-to-end data solutions, data pipelines, and ETL processes. With your 10+ years of experience in data engineering and data architecture, you will leverage your strong knowledge of data technologies such as Snowflake, Databricks, Apache Spark, Hadoop, Dbt, Fivetran, and Azure Data Factory. Your expertise in Python and SQL will be crucial in ensuring the success of various data projects. Key Responsibilities: - Design and develop end-to-end data solutions, data pipelines, and ETL processes - Utilize data technologies such as Snowflake, Databricks, Apache Spark, Hadoop, Dbt, Fivetran, Azure Data Factory - Ensure dat...

You will be joining the data engineering team as an experienced Python + Databricks Developer. Your role will involve designing, developing, and maintaining scalable data pipelines using Databricks and Apache Spark. You will write efficient Python code for data transformation, cleansing, and analytics. Collaboration with data scientists, analysts, and engineers to understand data needs and deliver high-performance solutions is a key part of this role. Additionally, you will optimize and tune data pipelines for performance and cost efficiency and implement data validation, quality checks, and monitoring. Working with cloud platforms, preferably Azure or AWS, to manage data workflows will also...

Role Overview: As a Developer contracted by Luxoft for supporting customer initiatives, your main task will involve developing solutions based on client requirements within the Telecom/network work environment. You will be responsible for utilizing technologies such as Databricks and Azure, Apache Spark, Python, SQL, and Apache Airflow to create and manage Databricks clusters for ETL processes. Integration with ADLS, Blob Storage, and efficient data ingestion from various sources including on-premises databases, cloud storage, APIs, and streaming data will also be part of your role. Key Responsibilities: - Develop solutions based on client requirements in the Telecom/network work environment...

Role Overview: You should be proficient in programming languages such as Python and R. You must have a strong understanding of machine learning algorithms and frameworks like TensorFlow, PyTorch, and scikit-learn. Experience with conversational AI platforms, especially Oracle Digital Assistant, is preferred. Additionally, you should have familiarity with data processing tools and platforms such as Apache Spark and Hadoop. Key Responsibilities: - Possess a strong understanding of machine learning algorithms and frameworks like TensorFlow, PyTorch, and scikit-learn - Be proficient in programming languages such as Python and R - Experience with conversational AI platforms, especially Oracle Dig...

As a Senior Azure Data Engineer, you will be responsible for designing, building, and optimizing scalable data pipelines and solutions using Databricks and modern Azure data engineering tools. Your expertise in Databricks and Azure services will be crucial in delivering high-quality, secure, and efficient data platforms. - Strong hands-on experience with Databricks - Proficiency in Azure Data Factory (ADF) for orchestrating ETL workflows - Excellent programming skills in Python with advanced PySpark skills - Solid understanding of Apache Spark internals and tuning - Expertise in SQL for writing complex queries and optimizing joins - Familiarity with data warehousing principles and modeling t...

Required Skills : Data Engineer, Azure, Azure Databricks, Azure Datafactory, Apache Spark, Data Modeling, Data Warehousing, BI tools, Git, Python, SQL

Share resume to : sowmya.v@acesoftlabs.com Hiring Now: Azure Data Engineer (36 Years) – Immediate Opportunity! Location : Bangalore Experience : 3–6 Years Key Responsibilities: Design and build scalable data solutions using Azure Data Factory, Azure Databricks, and Azure Synapse . Develop ETL/ELT pipelines for structured and unstructured data from various sources and formats. Write efficient SQL queries, stored procedures, functions , and dynamic SQL for data transformation and analysis. Implement data lake architecture and optimize data ingestion using Apache Spark with Scala or Python . Handle and maintain Delta Tables within the Databricks environment. Collaborate with DevOps team to supp...

Azure Data Tech Lead / Sr.Azure Data Engineer Years of Exp- 6 to 8 Years Location-Chennai Work Mode- Work From Office -Chennai Shift time-UK Shift time Billrate- Ieyond this range) Primary Skillset-Azure datafactory,Azure ,Python, Pyspark, Databricks, SQL,Azure Synapse JD: Experience with Azure datafactory,Azure ,Python, Pyspark, Databricks, SQL Key Responsibilities Design and build ETL/ELT pipelines to ingest, transform, and load data from clinical, omics, research, and operational sources. Optimize performance and scalability of data flows using tools like Apache Spark, Databricks, or AWS Glue. Collaborate with domain experts in genomics, clinical trials, and lab science to understand and ...

About the Role: We are hiring a Senior Data Engineer with strong expertise in Palantir Foundry or Gotham to lead the design, development, and optimization of data engineering solutions across large-scale enterprise environments. The ideal candidate will have deep experience building secure, high-performance data pipelines using Python, ETL/ELT practices, distributed computing frameworks, and cloud platforms ( AWS ,Azure ,GCP)while working closely with DevOps, analytics, and business teams to enable data-driven decisions. Key Responsibilities: Architect, build, and optimize large-scale ETL/ELT pipelines using Palantir Foundry/Gotham . Integrate diverse data sources including SQL , NoSQL , API...

Role Overview: At Toyota Connected, you will have the opportunity to transform the automotive industry and positively impact others on a global scale. As part of the Operations - Product Quality team, you will work in a collaborative and fast-paced environment to deliver exceptional value through connected products and services. Your focus will be on continual improvement and re-imagining what mobility can be today and in the future. Key Responsibilities: - Design, develop, and maintain automation test frameworks and scripts specifically for API-level testing. - Validate and verify complex Machine Learning applications, ensuring models and data pipelines perform accurately in production envi...

As a Machine Learning Engineer in this role, you will be responsible for working closely with customers, product teams, and engineering to achieve the following key responsibilities: - Onboard new clients and configure solutions to meet their data and business requirements. - Validate data quality and integrity to ensure accurate results. - Deploy and monitor machine learning models in production environments. - Execute existing ML pipelines for training new models and evaluating their quality. - Interpret model performance metrics and provide valuable insights to customers and internal teams. - Clearly communicate technical concepts to non-technical stakeholders. - Offer actionable feedback...

As an experienced software developer with expertise in .NET Core, Azure, DevOps, and Databricks, your role will revolve around designing, developing, and maintaining high-performance applications while collaborating with cross-functional teams to translate business requirements into technical solutions. Your responsibilities will include developing and integrating APIs to support seamless retail operations. - Application Development: - Design, develop, and maintain high-performance applications using .NET Core. - Collaborate with cross-functional teams to translate business requirements into technical solutions. - Develop and integrate APIs that support seamless retail operations. - Cloud Ar...

As a Mid/Senior Software Engineer at our company, you will play a crucial role in developing and maintaining the Data Management Platform within the Big Data Team. You will be an integral part of a cross-functional international Agile team consisting of talented engineers, product managers, and leaders across various locations globally. Your primary programming language will be Java/Scala, and the majority of your work will revolve around the Hadoop ecosystem. **Role Overview:** - As a Mid/Senior Software Engineer, you will have a significant impact on the team by contributing to the development and management of high scale/load distributed systems. Your work will provide essential data to k...

Key Skills & Responsibilities Hands-on experience with AWS services: S3, Lambda, Glue, API Gateway, and SQS. Strong data engineering expertise on AWS, with proficiency in Python, PySpark, and SQL. Experience in batch job scheduling and managing data dependencies across pipelines. Familiarity with data processing tools such as Apache Spark and Airflow. Ability to automate repetitive tasks and build reusable frameworks for improved efficiency. Provide RunOps DevOps support, and manage the ongoing operation and monitoring of data services. Ensure high performance, scalability, and reliability of data workflows in cloud environments. Skills: aws,s3,glue,apache spark,lambda,airflow,sql,s3, lambda...