530 Apache Spark Jobs - Page 21

We're Nagarro. We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale across all devices and digital mediums, and our people exist everywhere in the world (18000+ experts across 38 countries, to be exact). Our work culture is dynamic and non-hierarchical. We're looking for great new colleagues. That's where you come in! REQUIREMENTS: Total experience 6+years. Hands on experience in Spark, Python/Scala, Azure Databricks, Data Pipelines, SQL Server / NoSQL . Strong working knowledge in Python, Scala, SQL, Airflow and PySpark. We are looking for a Senior Data Engineer with strong...

Responsibilities: Design and articulate enterprise-scale data architectures incorporating multiple platforms including Open Source and proprietary Data Platform solutions - Databricks, Snowflake, and Microsoft Fabri c, to address customer requirements in data engineering, data science, and machine learning use cases. Conduct technical discovery sessions with clients to understand their data architecture, analytics needs, and business objectives Design and deliver proof of concepts (POCs) and technical demonstrations that showcase modern data platforms in solving real-world problems Create comprehensive architectural diagrams and i mplementation roadmaps for complex data ecosystems spanning c...

Responsibilities Lead a team of data engineers, providing technical mentorship, performance management, and career development guidance Design and oversee implementation of modern data architecture leveraging cloud platforms ( AWS, Azure, GCP ) and industry-leading data platforms (Databricks, Snowflake, Microsoft Fabric) Establish data engineering best practices, coding standards, and technical documentation processes Develop and execute data platform roadmaps aligned with business objectives and technical innovation Optimize data pipelines for performance, reliability, and cost-effectiveness Collaborate with data science, analytics, and business teams to understand requirements and deliver ...

Role :- Resident Solution ArchitectLocation: RemoteThe Solution Architect at Koantek builds secure, highly scalable big data solutions to achieve tangible, data-driven outcomes all the while keeping simplicity and operational effectiveness in mind This role collaborates with teammates, product teams, and cross-functional project teams to lead the adoption and integration of the Databricks Lakehouse Platform into the enterprise ecosystem and AWS/Azure/GCP architecture This role is responsible for implementing securely architected big data solutions that are operationally reliable, performant, and deliver on strategic initiatives Specific requirements for the role include: Expert-level knowled...

As a Principal Data Engineer on the Marketplace team, you will be responsible for analysing and interpreting complex datasets to generate insights that directly influence business strategy and decision-making. You will apply advanced statistical analysis and predictive modelling techniques to identify trends, predict future outcomes, and assess data quality. These insights will drive data-driven decisions and strategic initiatives across the organization. The Marketplace team is responsible for building the services where our customers will go to purchase pre-configured software installations on the platform of their choice. The challenges here are across the entire stack, from back-end dist...

Senior Data Engineer We are seeking a highly skilled Senior Data Engineer with strong expertise in big data, Spark, and modern cloud-native platforms. The ideal candidate will work closely with data architects, platform teams, and vendor products to build scalable, sustainable, and performant data pipelines and APIs in a dynamic offshore environment. Location: Offshore (Gurugram) Job specific Duties and Responsibilities Design, develop, and optimize data pipelines for processing large volumes of structured and unstructured data using Apache Spark and AWS technologies. Develop APIs and microservices using container frameworks like OpenShift, Docker, and Kubernetes. Work with diverse data form...

What you will do Lets do this. Lets change the world. In this vital role you will create and develop data lake solutions for scientific data that drive business decisions for Research. You will build scalable and high-performance data engineering solutions for large scientific datasets and collaborate with Research collaborators. You will also provide technical leadership to junior team members. The ideal candidate possesses experience in the pharmaceutical or biotech industry, demonstrates deep technical skills, is proficient with big data technologies, and has a deep understanding of data architecture and ETL processes. Roles & Responsibilities: Lead, manage, and mentor a high-performing t...

• Design and implement scalable, efficient and high-performance data pipelines • Develop and optimize ETL/ELT workflows using modern tools and frameworks. • Work with cloud platforms (AWS, Azure, GCP) Detailed JD will be given later.

Preferred candidate profile - Currently serving Notice Period who can join with in 15--20 days only ! 5+ Years of Experience is Mandate in AWS and Bigdata Role & responsibilities Experience with Big Data technologies will be a plus ( Hadoop, Spark, Kafka, HBase, etc ) • Write SQL queries to validate the dashboard output • Working experience with database environment - understanding relational database structure and hands-on SQL knowledge to extract/manipulate data for variance testing. • Performing code reviews and pair programming • Supporting and enhancing current applications • Design, develop, test, and implement the application investigate and resolve complex issues while supporting exi...

Role & responsibilities Description - External Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by diversity and inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health equity on a global scale. Join us to start Caring. Connecting. Growing together Primary Responsibilities: Design, develop, and maintain scala...

Overview: As a Data Engineer, you will work with multiple teams to deliver solutions on the AWS Cloud using core cloud data engineering tools such as Databricks on AWS, AWS Glue, Amazon Redshift, Athena, and other Big Data-related technologies. This role focuses on building the next generation of application-level data platforms and improving recent implementations. Hands-on experience with Apache Spark (PySpark, SparkSQL), Delta Lake, Iceberg, and Databricks is essential. Responsibilities: • Design and develop data lakes, manage data flows that integrate information from various sources into a common data lake platform through an ETL Tool • Code and manage delta lake implementations on S3 u...

Who are we looking for? We are looking for a data bricks engineer (developer), having strong software development experience of 6 to 10 years on Apache Spark & Scala Technical Skills: Strong knowledge & hands-on experience in Apache Spark & Scala Experience in AWS S3, Redshift, EC2 and Lambda services Extensive experience in developing and deploying Bigdata pipelines Experience in Azure data lake Strong hands on in SQL development / Azure SQL and in-depth understanding of optimization and tuning techniques in SQL with Redshift Development in Notebooks (like Jupyter, DataBricks, Zeppelin etc) Development experience in Spark Experience in scripting language like python and any other programmin...

We are seeking an experienced Java Technical Lead with 8-12 years of experience for an immediate joiner position in Noida. The role involves leading a technical team in the design, development, and implementation of scalable Java-based enterprise solutions using Agile methodologies. Responsibilities include defining architecture, capturing requirements, leveraging Spring Boot and Spring Cloud frameworks, performing code reviews, implementing PoCs, troubleshooting bottlenecks, and ensuring system scalability and performance. The candidate should possess strong expertise in distributed architecture, multithreading, RDBMS, and NoSQL databases, with added knowledge in Apache Spark, Data Science,...

Role: Scala Data Engineer Location: Bangalore (Whitefield) Mode: Hybrid (3 days WFO/week) Experience Range: 8 to 15 years Mandatory Skills: Scala (preferred over Java), SQL, Apache Spark Strong Data Structures and Algorithms Hands-on experience with data engineering on large-scale systems Exposure to cloud-native environments (Azure preferred) Preferred Skills: Hadoop, Kafka, Streaming (Structured Streaming/Event Hub) CI/CD tools: Git, Docker, Jenkins Experience with Medallion Architecture, Parquet, Apache Iceberg Orchestration tools (Airflow/Oozie), NoSQL DBs Data Governance tools: Alation, Collibra, Lineage, Metadata Responsibilities: Build and optimize scalable data pipelines using Spark ...

Job Title: Data Engineer GenAI Applications Company: Amzur Technologies Location: Hyderabad / Visakhapatnam / Remote (India) Experience: 48 Years Notice Period: Immediate to 15 Days (Preferred) Employment Type: Full-Time Position Overview We are looking for a skilled and passionate Data Engineer to join our GenAI Applications team. This role offers the opportunity to work at the intersection of traditional data engineering and cutting-edge AI/ML systems, helping us build scalable, cloud-native data infrastructure to support innovative Generative AI solutions. What We’re Looking For – Required Skills & Experience 4–8 years of experience in data engineering or related fields. Strong programmin...

ETL DeveloperData Modeling Tools like ErwinSnowflake, Oracle, Amazon RDS (Aurora, Postgres), DB2, SQL server and Casandra,Apache Sqoop, AWS S3, Hue, AWS CLI, Amazon EMR, Amazon MSK, Amazon Sagemaker, Apache SparkAutosys, SFTP, AirFlow

Strong programming experience in Scala . Hands-on experience in Apache Spark using Scala , with deep understanding of distributed computing concepts. Experience with Core Spark APIs and Spark Streaming APIs . Familiarity with HDFS, Hive, S3, MongoDB , and relational databases. Experience in creating Scala/Spark jobs for large-scale data processing. Sound exposure to ETL pipeline implementation, batch scheduling, and automation . Proficiency in Hive queries . Understanding of big data concepts and distributed systems . Experience with Unix commands and shell scripting . Familiarity with Java (basic and advanced) is preferred.

Apache Spark Admin 5+ years CBD Belapur – Navi Mumbai

Experience Required : 10+ years overall, with 5+ years in Kafka infrastructure management and operations. Must have successfully deployed and maintained Kafka clusters in production environments, with proven experience in securing, monitoring, and scaling Kafka for enterprise-grade data streaming. Overview : We are seeking an experienced Kafka Administrator to lead the deployment, configuration, and operational management of Apache Kafka clusters supporting real-time data ingestion pipelines. The role involves ensuring secure, scalable, and highly available Kafka infrastructure for streaming flow records into centralized data platforms. Role & responsibilities Architect and deploy Apache Kaf...

Python Developer with Flask Np Immediate to 30 days Location – Hyderabad JD: We are seeking a skilled Python Developer with hands on experience in building RESTful APIs using Flask The ideal candidate will be responsible for designing developing and maintaining scalable backend services and integrating them with frontend components and thirdparty services Required Skills Strong proficiency in Python Experience with Flask and Flask extensions FlaskRESTful FlaskSQLAlchemy etc Understanding of RESTful API design principles Familiarity with relational databases eg PostgreSQL MySQL and ORMs Experience with version control systems eg Git Experience with cloud platforms GCP Knowledge of containeriz...

Job Type: Contract Experience Level: 3+ Years Job Overview: We are seeking an experienced Data Engineer to join our dynamic team. As a Data Engineer, you will be responsible for designing, building, and maintaining data pipelines, processing large-scale datasets, and ensuring data availability for analytics. The ideal candidate will have a strong background in distributed systems, database design, and data engineering practices, with hands-on experience working with modern data technologies. Key Responsibilities: Design, implement, and optimize data pipelines using tools like Spark, Kafka, and Airflow to handle large-scale data processing and ETL tasks. Work with various data storage systems...

Job Description: We are hiring a Senior Data Engineer with strong expertise in Palantir Foundry or Gotham to lead the design, development, and optimization of data engineering solutions across large-scale enterprise environments. The ideal candidate will have deep experience building secure, high-performance data pipelines using Python, ETL/ELT practices, distributed computing frameworks, and cloud platforms ( AWS ,Azure ,GCP)while working closely with DevOps, analytics, and business teams to enable data-driven decisions. Key Responsibilities: Architect, build, and optimize large-scale ETL/ELT pipelines using Palantir Foundry/Gotham . Integrate diverse data sources including SQL , NoSQL , AP...

Job Title: Senior Data Engineer Location: Pune Job Type: Full-Time IIT Graduates only Interested candidates can share resume on gloriya.bnh@gmail.com Job Description: We are looking for a Senior Data Engineer with strong expertise in Azure Data bricks, PySpark, and distributed computing to develop and optimize scalable ETL pipelines for manufacturing analytics. The role involves working with high-frequency industrial data to enable real-time and batch data processing. Desired Skills and Qualifications • Bachelors or Masters degree in Computer Science, Information Technology, or a related field. • 4+ years of experience in data engineering, with a strong focus on cloud platforms such as Azure...

Experience Required : 10+ years overall, with 5+ years in Kafka infrastructure management and operations. Must have successfully deployed and maintained Kafka clusters in production environments, with proven experience in securing, monitoring, and scaling Kafka for enterprise-grade data streaming. Overview : We are seeking an experienced Kafka Administrator to lead the deployment, configuration, and operational management of Apache Kafka clusters supporting real-time data ingestion pipelines. The role involves ensuring secure, scalable, and highly available Kafka infrastructure for streaming flow records into centralized data platforms. Role & responsibilities Architect and deploy Apache Kaf...

Experience Required : 10+ years overall, with 5+ years in Kafka-based data streaming development. Must have delivered production-grade Kafka pipelines integrated with real-time data sources and downstream analytics platforms. Overview : We are looking for a Kafka Developer to design and implement real-time data ingestion pipelines using Apache Kafka. The role involves integrating with upstream flow record sources, transforming and validating data, and streaming it into a centralized data lake for analytics and operational intelligence. Role & responsibilities Develop Kafka producers to ingest flow records from upstream systems such as flow record exporters (e.g., IPFIX-compatible probes). Bu...