905 Apache Spark Jobs - Page 8

As a Staff Software Engineer in the Data Quality & Audience team at Warner Bros. Discovery, you will play a crucial role in maintaining data quality and facilitating data-driven decisions. Your technical expertise will be vital in developing systems that adhere to quality standards and in creating datasets using top-notch software engineering and data management practices. Your responsibilities will include: - Applying comprehensive knowledge of technology options within the Data Engineering ecosystem - Constructing systems and datasets while adhering to best practices and operational excellence - Grasping data warehousing concepts, ETL processes, and tools such as Apache Airflow and Talend ...

As an AI/ML Product Designer at our company, your role involves designing and building AI/ML products at scale to enhance customer Understanding & Sentiment analysis, recommend customer requirements, suggest optimal inputs, and improve the efficiency of processes. You will collaborate with product owners and business stakeholders to achieve these goals. Key Responsibilities: - Lead a team of junior and experienced data scientists - Lead and participate in end-to-end ML project deployments, including feasibility analysis, design, development, validation, and application of state-of-the-art data science solutions - Push the boundaries of data science applications by utilizing data mining, visu...

As a Big Data Developer at Viraaj HR Solutions, your role involves designing and implementing scalable Big Data solutions using Hadoop technologies. You will be responsible for developing and maintaining ETL processes to manage and process large data sets. Collaborating with data architects and analysts to gather requirements and deliver solutions will be a key part of your responsibilities. Additionally, you will optimize existing Hadoop applications for maximizing efficiency and performance. Key Responsibilities: - Design and implement scalable Big Data solutions using Hadoop technologies. - Develop and maintain ETL processes to manage and process large data sets. - Collaborate with data a...

As a Software Engineer at Google, you will be developing next-generation technologies that impact how billions of users connect, explore, and interact with information and each other. Your role will involve handling information at massive scales and contributing to various areas such as information retrieval, distributed computing, system design, networking, security, artificial intelligence, and more. You will have the opportunity to work on critical projects and collaborate with diverse teams, showcasing your leadership skills and adaptability as you tackle new challenges across the full technology stack. Key Responsibilities: - Enhance Apache Spark for improved performance, reliability, s...

As a senior Applications Development Technology Lead Analyst at our company, your role involves establishing and implementing new or revised application systems and programs in coordination with the Technology team. Your primary objective is to lead applications systems analysis and programming activities. Responsibilities: - Partner with multiple management teams to ensure appropriate integration of functions to meet goals and identify necessary system enhancements for deploying new products and process improvements - Resolve high impact problems/projects through in-depth evaluation of complex business processes and industry standards - Provide expertise in applications programming and ensu...

As a Data Engineer at Capgemini Invent, your role will involve designing, developing, and maintaining scalable data pipelines to support ML models. You will collaborate closely with data scientists and ML engineers to ensure high-quality data inputs for ML applications. Additionally, you will conduct exploratory data analysis, manage datasets of various data types, and optimize data workflows for efficiency and performance. Your key responsibilities will include: - Designing, developing, and maintaining scalable data pipelines for ML models. - Performing data preprocessing, cleansing, and labeling to ensure high-quality data inputs. - Conducting exploratory data analysis to uncover insights ...

You are a Senior Backend Engineer with 7+ years of experience sought by Commotion to build scalable backend systems. You should have proficiency in Java, Spring Boot, and Spring Data, along with experience in Elasticsearch, Kafka, Redis, and Apache Spark. Your expertise in multi-threaded and distributed systems, SQL and non-relational databases, and cloud platforms (AWS, GCP) is a plus. - Design and build scalable services using Java and related technologies for backend development. - Work with Elasticsearch, Kafka, and Redis for real-time data processing in data integration. - Optimize systems for scalability and performance using Apache Spark for performance optimization. - Collaborate wit...

In this role at Markovate, you will be responsible for designing, developing, and maintaining scalable data pipelines for ingestion, processing, and storage. Your tasks will involve optimizing data architectures and models, developing and maintaining ETL processes, ensuring data quality and security, collaborating with cross-functional teams, monitoring and troubleshooting data workflows, and implementing data integration solutions using tools like Apache Spark, Kafka, and Airflow. Additionally, you will be documenting data processes and flow diagrams for internal transparency. Key Responsibilities: - Design, develop, and maintain scalable data pipelines for ingestion, processing, and storag...

Role Overview: As a Senior Data Engineer at the company, you will be responsible for designing and maintaining scalable data solutions to meet analytical and business decision-making needs. You should possess a strong background in data architecture, pipeline development, and cloud technologies. Key Responsibilities: - Design, build, and maintain robust and scalable ETL pipelines for large-scale data processing. - Develop and optimize complex SQL queries for data extraction, transformation, and analysis. - Collaborate with data scientists, analysts, and other engineering teams to support data initiatives and ensure consistency across projects. - Model and structure data following data wareho...

As an Associate Data Scientist in Chennai, your role involves developing analytics models using specialized tools based on the business problem and available data. You are responsible for identifying the appropriate models, developing the necessary code or package to implement them, and evaluating the validity of the models both scientifically and from a business perspective. Additionally, you will support the Data Science Team in designing and executing analytics projects, working with stakeholders and subject matter experts to complete tasks and deliverables, and collaborating across multiple sets of stakeholders to meet project goals. Key Responsibilities: - Develop analytics models using...

Role & responsibilities Skills Proficiency Level (1 - 10) EXP in Months Technical Skills (Tools, Languages, Competencies) Data processing ML Ops apache spark Monitoring/Alerting ML Ops High (Prometheus, Grafana) Training Infrastructure ML Ops GPU Training/AWS/Azure Data Bases ML Ops Vector Databases Programming/ Scripting Language ML Ops Python ML Ops Pyspark

As a Data Engineer in this role, you will be responsible for designing and implementing systems using PySpark. Your expertise in Apache Spark and Python programming will play a crucial role in developing data processing tasks such as database migration, transformation, and integration solutions for Data warehousing projects. Your tasks will include reading data from external sources, merging data, performing data enrichment, and loading into target data destinations. Additionally, you will need to build APIs to provision data to downstream systems leveraging different frameworks. Familiarity with Jupyter Notebook, Zeppelin, and PyCharm IDEs will be advantageous for this position. Key Respons...

Role Overview: As a Software Engineer at Meltwater, you will be part of the Platform Engineering team dedicated to combating misinformation at scale. Your role will involve designing, building, and maintaining robust backend systems and data pipelines to support misinformation detection, reporting, and mitigation efforts. Key Responsibilities: - Build and maintain scalable and resilient backend systems using Python and/or JVM-based languages. - Collaborate with cross-functional teams, including data scientists, ML engineers, and product managers. - Design and implement data processing pipelines using tools like Apache Spark and Kafka. - Develop APIs and services for real-time misinformation ...

You will be a part of the global, multi-disciplinary team at HARMAN Digital Transformation Solutions (DTS), where innovation is at the forefront of technology to transform the future. Your role will involve combining physical and digital elements to create dynamic solutions, working with various technologies like cross channel UX, cloud, data insights, IoT, and mobility. You will empower companies to develop new digital business models, expand into new markets, and enhance customer experiences. - Design and implement scalable processing and storage of large datasets with Big Data System architecture skills - Utilize Scala/Python programming languages and work with AWS Services like S3, Lambd...

As a Data Engineer at the company, you will play a crucial role in designing, developing, and maintaining scalable and efficient big data processing pipelines for distributed computing systems. You will collaborate with cross-functional teams to understand data requirements and design appropriate data solutions. Your key responsibilities will include: - Implementing data ingestion, processing, and transformation processes to support various analytical and machine learning use cases. - Optimizing and tuning data pipelines for performance, scalability, and reliability. - Monitoring and troubleshooting pipeline performance issues, identifying and resolving bottlenecks. - Ensuring data quality a...

Role Overview: You will be responsible for driving the IT strategy to create value across the organization as a GCP Big Query Engineer at Tredence. Your role involves implementing both low-level, innovative solutions and day-to-day tactics to enhance efficiency, effectiveness, and value. Your analytical skills will be key in providing critical content for decision-making and successful collaboration with business stakeholders. Key Responsibilities: - Implement and architect solutions on the Google Cloud Platform (GCP) using components of GCP - Utilize Apache Beam, Google Dataflow, and Apache Spark to create end-to-end data pipelines - Work with technologies such as Python, Hadoop, Spark, SQL...

Role & responsibilities Lead and mentor a team of full stack engineers, fostering a culture of technical excellence and collaboration. Architect, design, and develop scalable and robust full stack applications using Python and GoLang for backend services and React.js for front-end development. Manage the entire software development lifecycle, including planning, coding, testing, deployment, and maintenance. Design and manage databases using PostgreSQL , ensuring data integrity and performance. Utilize and manage services on AWS (Amazon Web Services) , including EC2, S3, RDS, and others, to deploy and scale applications. Implement and manage data processing pipelines using technologies like A...

Job Title: Engineer Work Location: ~BANGALORE~ Skill Required - Digital : Amazon Web Service(AWS) Cloud Computing Experience Range - 6 to 8 Job Description Key Accountabilities: • Experience building data pipelines for various heterogenous data sources. • Identifying, designing and implementing scalable data delivery pipelines and automating manual processes • Building required infrastructure for optimal data extraction, transformation and loading of data using cloud technologies like AWS, Azure etc., • Develop end to end processes on the enterprise level for use by the clinical data configuration specialist to prepare data extraction and transformations of raw data quickly and efficiently f...

As a skilled PySpark Developer with hands-on experience in Reltio MDM, your role will involve designing and implementing scalable data processing solutions using PySpark and integrating with Reltio's cloud-native MDM platform. Key Responsibilities: - Develop and maintain data pipelines using PySpark in distributed computing environments such as AWS EMR and Databricks. - Integrate and synchronize data between enterprise systems and the Reltio MDM platform. - Design and implement data transformation, cleansing, and enrichment processes. - Collaborate with data architects, business analysts, and Reltio solution architects to ensure high-quality data modeling. - Work on API-based integration bet...

As an AI/ML Developer at our company, you will be responsible for designing, developing, and deploying state-of-the-art AI/ML models to address complex business challenges. You will work closely with cross-functional teams to convert business requirements into technical specifications for data-driven applications. Your role will involve implementing machine learning algorithms, fine-tuning existing models, and integrating AI/ML solutions into production environments. Furthermore, you will be expected to mentor junior developers, keep abreast of emerging AI technologies, and explore innovative solutions to enhance our product offerings. Key Responsibilities: - Design, develop, and deploy adva...

As a Senior Data Engineer, your role involves building and maintaining automated data pipelines to ensure smooth data flow. You will also be responsible for ensuring data integrity, validation, and transformation for accurate analysis. Working efficiently with large datasets and maintaining data quality standards will be crucial. Additionally, collaborating effectively with cross-functional teams to meet project requirements is an essential part of your responsibilities. Qualifications Required: - Strong hands-on experience in SQL (Postgres/Other RDBMS), Python for data processing, Apache Spark, and cloud-based ETL tools such as AWS Glue, Lambda, Step Functions, Azure Data Factory, or Databr...

As a Head of Analytics or Data Architect with over 15 years of experience, you will play a crucial role in designing and implementing an organizational data strategy aligned with business processes. Your responsibilities will include developing data models, managing data warehouses, implementing data governance standards, and collaborating with business leaders to define actionable metrics and data visualization dashboards. - Interact closely with Business Leaders and Product Team to design, develop, and implement an overall organizational data strategy - Identify data sources and build complex data ingestion, data integration, and ETL strategy - Create and maintain an end-to-end vision for ...

As an AI Architect specializing in Agentic AI and Generative AI, your role will involve designing, developing, and deploying cutting-edge autonomous AI systems. You will be responsible for building LLM-powered agents with memory, tool usage, planning, and reasoning capabilities to create intelligent, goal-oriented systems. Your technical leadership will oversee AI initiatives from research and architecture design to deployment in cloud environments. Key Responsibilities: - Architect and construct LLM-based agents for autonomous task execution, memory management, tool usage, and multi-step reasoning. - Develop modular, goal-oriented agentic systems utilizing tools like LangChain, Auto-GPT, Cr...

Role Overview: As a Senior Data Scientist, you will play a crucial role in leading data-driven projects and shaping the data science strategy of the company. Your responsibilities will include developing advanced algorithms, managing large datasets, and mentoring junior team members. Key Responsibilities: - Design and implement complex statistical models and machine learning algorithms. - Lead data science projects from inception to deployment, ensuring high-quality deliverables. - Provide technical mentorship and guidance to junior data scientists. - Collaborate with business stakeholders to define project objectives and offer actionable insights. - Explore and implement new technologies an...

Role Overview: As a Research Scientist at Adobe, you will play a crucial role in the Media and Data Science Research Laboratory within the Digital Experience business unit. Your primary responsibility will involve designing, implementing, and optimizing machine learning algorithms to address real-world problems. Your focus will be on understanding user behavior, enhancing marketing performance through content generation, and developing scalable synthetic data generation techniques. Additionally, you will collaborate with various teams to prototype and transition research concepts into production. Key Responsibilities: - Design, implement, and optimize ML algorithms for understanding user beh...