1537 Apache Spark Jobs - Page 2

The role is responsible for developing and maintaining the data architecture of the Enterprise Data Fabric. Data Architecture includes the activities required for data flow design, data modeling, physical data design, query performance optimization. The Data Modeler position is responsible for developing business information models by studying the business, our data, and the industry. This role involves creating data models to realize a connected data ecosystem that empowers consumers. The Data Modeler drives cross-functional data interoperability, enables efficient decision-making, and supports AI usage of Foundational Data. Roles & Responsibilities: Develop and maintain conceptual logical,...

What you will do In this vital role We are looking for highly motivated expert Senior Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibilities: Design, develop, and maintain scalable ETL/ELT pipelines to support str...

Job Title Consultant Data Engineer (AWS) Company Name Top MNCs(Payroll :: Waterleaf Consultants Pvt. Ltd.) Work Type Contract(6 Months) Job Location Hyderabad (Work From Office) Experience Required 8-12 Years (Minimum 8+ years of hands-on Data Engineering experience) Shift Day Shift | Monday to Friday Job Description We are hiring an experienced Consultant Data Engineer to design, build, and maintain scalable and secure data pipelines on AWS. The role involves working closely with Data Scientists, Analysts, and IT teams to deliver high-quality data solutions for analytics, reporting, and machine learning use cases. Key Responsibilities Design, build, and maintain scalable ETL/ELT data pipeli...

Role Overview: You will be joining the ML Solutions team within Markets OPS Technology as an ML Systems Engineer with ML Ops experience. Your primary responsibility will be to build and maintain scalable backend systems, deploy data science models, and design robust pipelines for seamless integration of DS solutions into applications. You will collaborate closely with data scientists and front end engineers to bridge the gap between data science and production-ready systems. Key Responsibilities: - ML backend and Orchestration: - Develop ML Systems backend infrastructure, messaging, and integration with interfacing systems - Design, build, and maintain APIs, Microservices, and systems to ser...

As a Software Engineer at Fanatics, you will have the opportunity to work on cutting-edge projects related to data platforms and privacy compliance. You will play a crucial role in designing and implementing components of the consent management framework, building privacy compliance reporting features, and contributing to the data certification framework. Additionally, you will work on developing data lineage tracking capabilities and maintaining APIs for integrating data quality, privacy, and governance tools. Your responsibilities will also include implementing data classification and privacy-preserving transformation pipelines, collaborating with cross-functional teams, participating in c...

As an Associate Data Scientist in Chennai, your role involves developing analytics models using specialized tools based on the business problem and available data. You will be responsible for identifying the right set of models, developing the necessary code/packages, and evaluating the validity of the models from both scientific and business perspectives. Additionally, you will support the Data Science Team in designing and executing analytics projects, collaborating with stakeholders and subject matter experts to deliver on project tasks and deliverables. Key Responsibilities: - Develop analytics models using specialized tools - Identify and develop the right set of models - Evaluate model...

Role Overview: Capgemini Invent is looking for a Data Engineer with 5+ years of experience in data engineering to join their team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines to support ML models. You will work closely with data scientists and ML engineers to ensure high-quality data inputs for ML applications. Additionally, you will be involved in data preprocessing, cleansing, exploratory data analysis, and managing datasets comprising text, image, audio, and video data. Key Responsibilities: - Design, develop, and maintain scalable data pipelines for ML models - Perform data preprocessing, cleansing, and labeling - Conduct...

You are a highly skilled Java Backend Developer with over 7 years of experience, including at least 6 years of relevant experience. You have expertise in Core Java, J2EE, Spring Boot, Microservices, and Apache Spark. Your role involves designing, developing, and maintaining scalable backend systems using modern cloud and data technologies. - Design and develop backend services using Java, J2EE, Spring Boot, and Microservices architecture. - Build and optimize RESTful APIs and backend components for high performance and reliability. - Work extensively with Apache Spark for data processing and integration. - Collaborate with cross-functional teams to design scalable and resilient solutions. - ...

As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution. Your primary responsibilities include: Lead the design and construction of new solutions using the latest technologies, always looking to add business value and meet user requirements. Strive for continuous improvements by testing the build solution and working under an agile framework. Discover and implement the latest technologies trends to maximize and build creative solutions Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Ex...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and AWS Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark, Scala, and Hive, Hbase or other NoSQL databases on Cloud Data Platforms (AWS) or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developin...

About the Role: The charter of the Data + ML Platform team is to harness all the data that is ingested and cataloged within the Data LakeHouse for exploration, insights, model development, ML Engineering and Insights Activation. This team is situated within the larger Data Platform group, which serves as one of the core pillars of our company. We process data at a truly immense scale. Our processing is composed of various facets including threat events collected via telemetry data, associated metadata, along with IT asset information, contextual information about threat exposure based on additional processing, etc. These facets comprise the overall data platform, which is currently over 200 ...

We are seeking a highly skilled and experienced Spark Cluster Admin to join our dynamic team. This role is critical for building, optimizing, and supporting our cutting-edge data platform, leveraging Apache Spark , Apache Iceberg , and a robust DevOps approach within an OpenShift environment. The ideal candidate will be adept at both developing high-performance data solutions and ensuring their stability and reliability in a production setting. Key Responsibilities Design, develop, and optimize scalable and resilient data processing applications using Apache Spark (batch, streaming, and real-time). Implement and manage data pipelines, ensuring data quality, consistency, and performance. Perf...

As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: * Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. * Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. * Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Required education ...

Boeing India is currently seeking an Associate Data Engineer to join the product development team. The development team provides comprehensive software solutions to rapidly access and visually transform complex engineering and manufacturing product data. Responsibilities include development and maintenance of in-house software applications supporting our business lines. Good communication skills are needed both verbally and written, to interact with peers and customers. Job requires working within a diverse team of skilled and motivated co-workers to collaborate on results. Other qualities for this candidate are a positive attitude, self-motivated, the ability to work in a fast-paced, demand...

The Team: We are looking for a highly motivated, enthusiastic, and skilled engineering leader for S&P Global Energy. We strive to deliver solutions that are sector-specific, data-rich, and hyper-targeted for evolving business needs. The Impact: S&P Global Energy is seeking a Lead Software Developer to provide technical leadership for all stages of software design, development, and maintenance for data processing applications. This person would be part of a development team that manages and supports the internal applications that support Horizon business datasets. This role requires a candidate to handle data processing and develop applications for big data. We have teams composed of individu...

The Opportunity As an Associate Architect in the Machine Learning Platform team, you will be a key technical leader in the Machine Learning Engineering team. You will be responsible for the strategic vision, design, and implementation of Myntras end-to-end ML platform. This is a high-impact role for a seasoned ML practitioner who has transitioned into an architectural role. You will draw upon your deep, hands-on experience as a Machine Learning Engineer to tackle complex, large-scale challenges and shape the future of how Myntra leverages AI. You will work with a diverse set of ML applications, including computer vision for product tagging, recommendation engines, search ranking, and the new...

Role Description: As a Senior Data Engineer at Amgen, you will be responsible for managing and optimizing the company''s data infrastructure and architecture You will design and implement data pipelines, develop data models, perform data integration, and ensure data quality and governance Your expertise in data engineering, big data technologies, and data manipulation will contribute to the effective storage, processing, and utilization of large-scale data sets Key Responsibilities: Design, develop, and maintain scalable ETL/ELT pipelines to support structured, semi-structured, and unstructured data processing across the Enterprise Data Engineering for Biotech or Pharma functional knowledge ...

The role is responsible for designing, building, maintaining, analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing reports, supporting and executing data governance initiativesand visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has strong technical skills, experience with big data technologies, and a deep understanding of data architecture and ETL processes Roles & Responsibilities: Design, develop, and maintain data solutions for data generation, collection, and processing Be a key team member that assists in design and development of the...

We are seeking a Senior Technical Pre-Sales Engineer with strong hands-on background in enterprise systems, data platforms, and AI/ML solutioning for Manufacturing, Logistics, and Supply Chain clients.This role requires deep technical articulation , not just sales support. The candidate must be able to whiteboard architectures , challenge client assumptions, define scalable IT service delivery models , and translate operational pain points into data-driven, AI-enabled transformation solutions . Core Technical Responsibilities 1. Deep Technical Discovery & Domain Mapping Conduct architecture-level discovery sessions with enterprise and mid-market clients. Analyze existing systems such as: ERP...

What you will do In this vital role you will design, build and maintain data lake solutions for scientific data that drive business decisions for Research. You will build scalable and high-performance data engineering solutions for large scientific datasets and collaborate with Research stakeholders. The ideal candidate possesses experience in the pharmaceutical or biotech industry, demonstrates strong technical skills, has experience with big data technologies, and understands data architecture and ETL processes Design, develop, and implement data pipelines, ETL/ELT processes, and data integration solutions Contribute to data pipeline projects from inception to deployment, manage scope, tim...

Role Description: We are looking for highly motivated expert Principal Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks with domain expertise in R&D domain. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibilities: Architect and maintain robust, scalable data pipel...

You have an exciting opportunity to join a staffing and HR services firm focused on placing talent into Enterprise IT, Analytics, and Big Data engineering projects across various sectors. As an on-site Big Data Engineer in India, your role will involve designing, building, and optimizing data pipelines and platform components for critical data applications. **Role Overview:** As a Big Data Engineer, you will be responsible for: - Designing and implementing end-to-end batch and streaming data pipelines using Spark and Hadoop ecosystem components. - Developing ETL jobs and data transformation logic in Python/SQL to ensure scalability and maintainability. - Integrating and processing real-time ...