905 Apache Spark Jobs - Page 17

You are seeking a Tech Lead with expertise in Java, Apache Spark, and AWS (EMR, Lambda, EKS, Airflow) to design and deploy scalable cloud-based data solutions. Your role will involve leading big data pipeline development, serverless applications, microservices, and workflow orchestration. You will be responsible for mentoring the team, driving technical excellence, and ensuring the successful implementation of data engineering projects. Key Responsibilities: - Lead big data pipeline development using Java and Apache Spark - Design and deploy scalable cloud-based data solutions on AWS (EMR, Lambda, EKS, Airflow) - Implement serverless applications and microservices architecture - Orchestrate ...

Role Overview: At Toyota Connected, you will have the opportunity to change the way the world works, transform the automotive industry, and positively impact others on a global scale. In a collaborative and fast-paced environment, you will focus on continual improvement and work iteratively to deliver exceptional value through connected products and services that amaze and delight customers worldwide. Join us in re-imagining what mobility can be today and for years to come! Key Responsibilities: - Design, develop, and maintain automation test frameworks and scripts specifically for API-level testing. - Validate and verify complex Machine Learning applications, ensuring accurate performance o...

Key Responsibilities : Lead the design, development, and deployment of end-to-end data solutions on Azure Databricks platform. Work with data scientists, data engineers, and business analysts to design and implement data pipelines and machine learning models. Develop efficient, scalable, and high-performance data processing workflows and analytics solutions using Databricks , Apache Spark , and Azure Synapse . Manage and optimize Databricks clusters and data pipelines. Collaborate with cross-functional teams to gather requirements and deliver optimal solutions. Design and implement ETL processes using Databricks Notebooks, Azure Data Factory, and other Azure services. Ensure high availabilit...

Contractual Hiring manager :- My profile :- linkedin.com/in/yashsharma1608 Payroll of :- https://www.nyxtech.in/ 1. AZURE DATA ENGINEER WITH FABRIC The Role : Lead Data Engineer PAYROLL Client - Brillio About Role: Experience 6 to 8yrs Location- Bangalore , Hyderabad , Pune , Chennai , Gurgaon (Hyderabad is preferred) Notice- 15 days / 30 days. Budget -15 LPA AZURE FABRIC EXP MANDATE Skills : Azure Onelake, datapipeline , Apache Spark , ETL , Datafactory , Azure Fabric , SQL , Python/Scala. Key Responsibilities: Data Pipeline Development: Lead the design, development, and deployment of data pipelines using Azure OneLake, Azure Data Factory, and Apache Spark, ensuring efficient, scalable, and...

Are you a hands-on Cloud Architect with deep DevOps expertise We are seeking a highly skilled professional to lead our cloud migration journey (from ETL DataStage), drive automation implementation, and ensure cloud security at scale. The ideal candidate should possess expertise in Cloud Architecture (AWS / Azure / GCP), Apache Spark & Snowflake, CI/CD, Automation, Infrastructure as Code, DevOps best practices, and cloud security. Additionally, experience in ETL Migration, particularly DataStage, is preferred. This position is open for individuals with a minimum of 7 years of experience in Cloud architecture, 4 years in DevOps, and 3 years in Snowflake. The job type is contractual/temporary w...

You are a Python Developer with expertise in SQL and Spark, having 6 to 10 years of experience. You should be able to commence work immediately or within a notice period of up to 15 days. The job is offered on a contract basis through CoupA (via Partner) and the primary work locations include Bangalore, Hyderabad, and Pune. Your role requires strong hands-on coding skills in Python, a deep understanding of SQL including syntax and performance optimization, as well as prior experience working with Apache Spark. Familiarity with Cloud Platforms such as AWS, Azure, or GCP is also expected. This is a contractual position for a duration of 6 months with the requirement to work in person at the de...

As a GCP Data Engineer, you will play a crucial role in the development, optimization, and maintenance of data pipelines and infrastructure. Your proficiency in SQL and Python will be pivotal in the management and transformation of data. Moreover, your familiarity with cloud technologies will be highly beneficial as we strive to improve our data engineering processes. You will be responsible for building scalable data pipelines. This involves designing, implementing, and maintaining end-to-end data pipelines to efficiently extract, transform, and load (ETL) data from various sources. It is essential to ensure that these data pipelines are reliable, scalable, and performance-oriented. Your ex...

As a Big Data Engineer at our company, you will be an integral part of our data engineering team. Your primary responsibility will be to design, develop, and optimize scalable data pipelines and big data solutions. To excel in this role, you should have hands-on experience with the Hadoop ecosystem, Apache Spark, and programming languages such as Python (PySpark), Scala, and Java. Your expertise in these technologies will enable you to support analytics and business intelligence initiatives effectively. By leveraging your skills, you will contribute to the success of our data-driven projects and help drive insights that can positively impact our business. If you are a highly skilled and moti...

As a Data Engineer with expertise in Generative AI, you will play a crucial role in our dynamic team based in Jaipur. Your primary responsibility will be to design, develop, and maintain robust data pipelines and workflows, integrating Generative AI models to enhance existing data systems and drive innovative business outcomes. You will collaborate with cross-functional teams to understand business needs, translating them into scalable data and AI solutions while ensuring data security, quality, and governance across all processes. Additionally, you will stay updated with the latest advancements in Generative AI and data engineering practices. To excel in this role, you should possess a Bach...

We are seeking an experienced Java Technical Lead with 812 years of experience for an immediate joiner position in Noida. The role involves leading a technical team in the design, development, and implementation of scalable Java-based enterprise solutions using Agile methodologies. Responsibilities include defining architecture, capturing requirements, leveraging Spring Boot and Spring Cloud frameworks, performing code reviews, implementing PoCs, troubleshooting bottlenecks, and ensuring system scalability and performance. The candidate should possess strong expertise in distributed architecture, multithreading, RDBMS, and NoSQL databases, with added knowledge in Apache Spark, Data Science, ...

Role & responsibilities We are hiring an AI/ML Developer (India), to join our India team, in support of a large global client! You will be responsible for developing, deploying, and maintaining AI and machine learning models. Your expertise in Python, cloud services, databases, and big data technologies will be instrumental in creating scalable and efficient AI applications. What You Will Be Doing Develop, train, and deploy machine learning models for predictive analytics, classification, and clustering. Implement AI-based solutions using frameworks such as TensorFlow, PyTorch, and Scikit-learn. Work with cloud platforms including AWS (SageMaker, Lambda, S3), Azure, and Google Cloud (Vertex ...

Data Management: Proficiency in data architectures such as data warehouses, data lakes, and data hubs, along with supporting processes like data integration, governance, and metadata management DevOps Practices Required Candidate profile Data Pipeline Construction: Designing, building, 5 years of experience in Python, Apache Spark, DBT(1-2 Years), AWS (EMR, LAMBDA, EKS, EC2, RDS, S3, VPC) and Snowflake.

We are looking for a highly motivated and enthusiastic Intermediate Software Developer to join our growing engineering team. This role is suitable for an individual with 3-5 years of experience who is eager to learn and excel within a fast-paced environment. You will have the opportunity to work on exciting projects involving large-scale data processing, analytics, and software development, utilizing technologies such as Java, Apache Spark, Python, and Apache Iceberg. This position provides a unique chance to gain hands-on experience with cutting-edge data lake technologies and play a vital role in enhancing critical data infrastructure. As an Intermediate Software Developer, your responsibi...

As an individual contributor in the Search, Discovery, and Content (SDC) team at Adobe, you will play a crucial role in bridging the gap between development and operations teams. Your primary responsibility will be to foster collaboration, manage infrastructure availability, and enhance software delivery processes. The SDC team's platform supports search, browse, recommendation, and discovery functionalities in various Adobe Cloud products. This platform caters to millions of users and integrates with numerous products, requiring a focus on innovative initiatives such as NoSQL databases, Machine Learning, and other evolving data technologies. Your key responsibilities will include: - Working...

As a passionate Software Engineer with a proven track record of solving complex problems and driving innovation, you will have the opportunity to work at ReliaQuest and be part of a team that is reshaping the future of threat detection and response. Your role will involve developing cutting-edge security technology, creating REST APIs, and integrating various products to enhance our customers" threat detection capabilities. By working closely with talented individuals, you will contribute directly to the growth and success of our organization. Your responsibilities will include researching and developing solutions across a range of advanced technologies, managing deployment processes, conduc...

As a member of the development team at SAP, you will have the opportunity to collaborate with highly technical and motivated developers. Your role will involve working closely with stakeholders to understand business requirements and translate them into technical solutions. You will be responsible for designing and implementing scalable and efficient code, following best practices and industry standards. Throughout the software development lifecycle, you will take ownership of your code from ideation to customer support. Additionally, you will build a comprehensive understanding of features across various components and adhere to agile engineering practices and processes followed by the team...

As a Senior Data Engineer at Barclays, you will play a crucial role in driving innovation and excellence by spearheading the evolution of the digital landscape. Your primary responsibility will be to harness cutting-edge technology to revolutionize digital offerings and ensure unparalleled customer experiences. You will be evaluated based on your proficiency in key critical skills such as experience with Modern Data Engineering Frameworks, developing and maintaining cloud-native applications, and job-specific skill sets. To excel in this role, you should possess the following qualifications and experience: - A Bachelor's or Master's degree in Computer Science, Engineering, or a related field...

You will be working at Paras Twin Tower, Gurgaon as a full-time employee for Falcon, a Series-A funded cloud-native, AI-first banking technology & processing platform. Falcon specializes in assisting banks, NBFCs, and PPIs to efficiently launch cutting-edge financial products like credit cards, credit lines on UPI, prepaid cards, fixed deposits, and loans. Since its inception in 2022, Falcon has processed over USD 1 billion in transactions, collaborated with 12 of India's top financial institutions, and generated revenue exceeding USD 15 million. The company is supported by prominent investors from Japan, the USA, and leading Indian ventures and banks. To gain more insights about Falcon, vis...

As a Data Engineer/Big Data Engineer with over 5 years of experience, you will be responsible for leveraging your strong expertise in Apache Spark with Scala or Java. Your role will involve hands-on experience in building Microservices-based architectures and proficiency in designing and managing ETL/ELT data pipelines. You will be expected to have experience with relational and NoSQL databases, along with familiarity with data analytics platforms and real-time data processing. Additionally, any experience with cloud platforms such as AWS, Azure, or GCP would be considered a plus in this role.,

Unlock your potential with Dassault Systmes, a global leader in Scientific Software Engineering as a Big Data Engineer in Pune, Maharashtra! Role Description & Responsibilities: - Data Pipeline Development: Design, develop, and maintain robust ETL pipelines for batch and real-time data ingestion, processing, and transformation using Spark, Kafka, and Python. - Data Architecture: Build and optimize scalable data architectures, including data lakes, data marts, and data warehouses, to support business intelligence, reporting, and machine learning. - Data Governance: Ensure data reliability, integrity, and governance by enabling accurate, consistent, and trustworthy data for decision-making. - ...

As a Spark Scala Developer, you will be responsible for designing and developing scalable data pipelines using Apache Spark and Scala. Your role will involve optimizing and troubleshooting Spark jobs for performance, working with massive datasets in on-prem Hadoop clusters or cloud platforms, and writing clean, modular Scala code using functional programming principles. Collaborating with data teams to integrate with platforms like Snowflake, Databricks, or data lakes will also be a key part of your responsibilities. It will be essential to ensure code quality, documentation, and CI/CD practices are followed diligently. The ideal candidate for this position should possess 3+ years of experie...

You are an experienced Data Architect who will be responsible for leading the transformation of enterprise data solutions, particularly focused on migrating Alteryx workflows into Azure Databricks. Your expertise in the Microsoft Azure ecosystem, including Azure Data Factory, Databricks, Synapse Analytics, Microsoft Fabric, and strong background in data architecture, governance, and distributed computing will be crucial for this role. Your strategic thinking and hands-on architectural leadership will ensure the development of scalable, secure, and high-performance data solutions. Your key responsibilities will include defining the migration strategy for transforming Alteryx workflows into sc...

As a member of our team at SAP, you will have the opportunity to collaborate with a group of highly technical and motivated developers. Your role will involve working with teams across different geographies and time zones, ensuring effective collaboration and communication. By engaging with stakeholders, you will gain insights into business requirements and translate them into technical solutions. Your responsibilities will include designing and implementing scalable and efficient code, adhering to best practices and industry standards. Throughout the software development lifecycle, from ideation to customer support, you will take ownership of your code and ensure its quality. In this role, ...

You are being offered a unique opportunity to join the Mastercard Economics Institute (MEI) as an Associate Economics Analyst in the Global Growth and Operations team. In this role, you will report to the Director, Growth and Operations and be responsible for blending advanced economic research with strong programming and data visualization skills. As an Associate Economics Analyst at MEI, you will play a crucial role in supporting client and stakeholder engagements across the institute. You will collaborate with a diverse team of economists, econometricians, developers, visualization experts, and industry partners. Your responsibilities will include developing and testing hypotheses at the ...

We are seeking a candidate with over 7 years of experience in the role of a Data Engineer. The ideal candidate should have proficiency in using various software/tools, including Spark, Kafka, Spark-Streaming, Azure SQL, relational SQL, NoSQL databases, Synapse Pipeline, Azure Data Factory, Databricks, Synapse Analytics, Azure Function, ADLS, Scala, Python, and Terraform. Responsibilities of an ETL Data Engineer include designing and maintaining optimal data pipeline architecture, creating complex data sets to meet business requirements, implementing process improvements, building infrastructure for data extraction and loading, developing analytics tools for actionable insights, collaborating...