1537 Apache Spark Jobs - Page 4

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

Description : Industry & Sector : Technology consulting firm operating in the blockchain, fintech and distributed ledger technology (DLT) space. We deliver enterprise-grade DLT integrations, ledger analytics and data engineering solutions for global clients, transforming on-chain and off-chain data into actionable business insights. Role : Senior Data Engineer (DLT) Remote (India). Role & Responsibilities : - Design and build scalable ETL/ELT pipelines to ingest, process and transform on-chain and off-chain ledger data using streaming and batch frameworks. - Integrate with DLT nodes, RPC endpoints and indexers to capture transactions, blocks and smart-contract events into analytical stores. ...

As a seasoned Senior Developer & Tech Lead, you will play a crucial role in writing clean and efficient code, building scalable systems, driving engineering excellence, and leading a team of talented developers in a fast-paced, Agile environment. Your deep hands-on experience in Java and Apache Spark, combined with a strong foundation in object-oriented design principles, will be essential for this role. **Key Responsibilities:** - Perform detailed impact analysis for code changes with understanding of dependencies across the application components - Design and develop scalable, high-performance code using Java and Bigdata / Apache Spark - Write high-quality, maintainable code that is modula...

As a Databricks Engineer at our company, your role involves designing, developing, and implementing big data solutions using Databricks and Apache Spark. You will collaborate with cross-functional teams to gather requirements, troubleshoot and resolve big data processing issues, and optimize solutions for performance and scalability. Your responsibilities will also include developing and maintaining Databricks notebooks, jobs, and workflows, as well as ensuring seamless integration of big data solutions. Additionally, you will be expected to develop and maintain technical documentation. Key Responsibilities: - Design, develop, and test big data solutions using Databricks and Apache Spark. - ...

As a senior member of the team responsible for developing large-scale, highly available, and fault-tolerant next-generation SaaS solutions purpose-built for Health Care Analytics, your role involves the following: - Core Technical Skills Ecosystem: - Advanced Java development with extensive experience in Java 21 features. - Deep understanding of Spring Framework, Spring Boot, and Spring Security. - Expertise in implementing authentication, authorization, and secure coding practices. - Implement robust security measures using Spring Security. - Proficient in JBoss BPM Suite (jBPM) for business process automation. - Experience with microservices architecture. - Python & R Development (Good To ...

As an experienced Backend Developer, you will be responsible for architecting and building complex, scalable, and high-performance backend systems. Your role will involve owning the end-to-end development and delivery of features and enhancements. You will design and optimize large-scale distributed data processing using Apache Spark and Nifi. Collaborating with product managers, architects, and other stakeholders to align on technical vision will be a crucial part of your responsibilities. Additionally, you will lead code reviews and ensure adherence to best practices and high code quality. Mentoring and guiding junior and mid-level engineers will also be part of your role. Key Responsibili...

As a highly experienced Azure Data Engineer with over 8 years of experience, your role will involve designing and implementing scalable data pipelines and ETL processes using Azure Data Factory (ADF), PySpark, and Databricks. You will be responsible for managing and optimizing Azure Data Lake and integrating it with Azure Synapse Analytics for large-scale data storage and analytics. Collaborating with cross-functional teams, you will gather requirements, design data solutions, and deliver actionable insights. Additionally, you will be developing and optimizing SQL queries for data extraction and transformation, applying data modeling techniques, and implementing best practices for data gover...

**Role Overview:** You will be responsible for integrating data from various siloed systems into CDF using interfaces, extractors, and SDKs. Additionally, you will use ETL tools like Azure Data Factory or Informatica PowerCenter for data integration. **Key Responsibilities:** - Utilize ETL tools such as Azure Data Factory or Informatica PowerCenter for data integration - Integrate data from different systems into CDF using interfaces, extractors, and SDKs - Program in languages like Python, especially for using the Cognite Python SDK - Work with REST APIs for data extraction and integration - Manage databases such as PostgreSQL and utilize database gateways - Extract data from cloud storage ...

As a member of the Data Core Apps Team at Agoda, you will be part of a dynamic department that plays a critical role in the company's data-related requirements. Your ultimate goal will be to enable and increase the use of data through creative approaches and implementation of powerful resources. You will be joining a diverse team from around the world, united by a passion to make an impact in the travel industry. **Role Overview:** In this role, you will primarily focus on building first-class big data tools to assist thousands of users working with big data systems daily. Your responsibilities will include working on Spark runtime/utils, data exchange system, ETL tools, data lineage system,...

Role Overview: As a Full Stack AI Developer at GlobalLogic, you will be responsible for developing and implementing AI solutions using a variety of programming languages and tools. You will work on projects that have a significant impact on clients around the world, contributing to innovative solutions and pushing the boundaries of what is possible in the AI field. Key Responsibilities: - Must Have: Proficiency in programming languages such as Python, Java/Scala - Must Have: Experience with data processing libraries like Pandas, NumPy, and Scikit-learn - Must Have: Proficient in distributed computing platforms such as Apache Spark (PySpark, Scala) and Torch - Must Have: Ability to develop AP...

Role Overview: You will be working as a GCP Data Engineer at StatusNeo, where you will be responsible for designing, developing, and maintaining scalable data pipelines and architectures on Google Cloud Platform. Your role will involve collaborating with data scientists, analysts, and other stakeholders to ensure that data systems are optimized for performance, reliability, and scalability. Key Responsibilities: - Design and implement data pipelines using GCP services such as BigQuery, Dataflow, Pub/Sub, and Cloud Storage. - Utilize tools like Apache Beam, Apache Spark, and Dataproc for data ingestion, processing, and transformation. - Manage and optimize cloud-based data storage solutions, ...

As a Solution Architect with 7+ years of experience, your role will involve designing, implementing, and optimizing enterprise-scale data platforms with a focus on Databricks. You will be expected to leverage your expertise in big data, cloud-native architectures, and advanced analytics to translate business requirements into scalable solutions while engaging stakeholders effectively. Key Responsibilities: - Architect and deliver end-to-end data solutions using Databricks features like Delta Lake, MLflow, and Unity Catalog. - Design and optimize data pipelines for both batch and real-time processing utilizing technologies such as Spark, Kafka, and Event Hubs. - Collaborate closely with busin...

As an AWS Data Engineer at YASH Technologies, you will play a crucial role in designing, developing, testing, and supporting data pipelines and applications using various AWS services. Your primary skillsets should include AWS services such as Glue, Pyspark, SQL, Databricks, and Python. Additionally, proficiency in ETL tools, Github, and DevOps (CI/CD) will be valuable secondary skillsets. Key Responsibilities: - Design, develop, test, and support data pipelines and applications using AWS services like Glue, Pyspark, SQL, Databricks, and Python. - Utilize your expertise in data integration and pipeline development, with a focus on AWS Cloud technologies. - Work with a mix of Apache Spark, Gl...

As an Engineer with 5-8 years of experience, you will be responsible for the following: - Engineer having competencies in Apache Beam, GCP Cloud, and Open Shift - Experience in Java and Kafka Streaming; knowledge of GCP and Kubernetes is a plus - Developing and deploying data and analytics-led solutions on GCP - Designing highly available and scalable systems - Hands-on experience with Data Engineering solutions using Cloud BigQuery, Cloud DataFlow, Cloud BigTable, Storage, Cloud Spanner, Cloud IAM - Understanding of Apache Kafka - Proficiency in cloud-based ETL/Data Orchestration tools like Apache Beam and Cloud Composer - Experience with Stack driver logging/monitoring - Proficient in Pyth...

As a skilled Technology Stack Administrator, your role will involve managing and optimizing the on-premises infrastructure which includes RedHat OpenShift, Apache Spark, Apache Airflow, Delta Lake, Apache Kafka, and Debezium. Your responsibilities will encompass a variety of tasks such as platform administration, monitoring & optimization, security & compliance, automation & CI/CD, troubleshooting & support. Key Responsibilities: - Platform Administration: - Install, configure, and maintain RedHat OpenShift clusters in an on-premises environment. - Build and orchestrate data pipelines using Apache Airflow. - Implement and manage Delta Lake for ACID-compliant data lake architecture. - Manage ...

Job Role Strong SQL and Python coding skills. Experience with distributed data processing frameworks (e.g., Apache Spark, Dask, Ray) in an enterprise setting. Familiarity with deep learning frameworks such as TensorFlow, PyTorch, or Keras. Proficient in PySpark for Machine Learning. Desired Skills Docker, Machine Learning, Python, MySQL DBA Qualifications Bachelor of Engineering

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Apache Spark, Microsoft SQL Server, Python (Programming Language), Databricks Unified Data Analytics Platform Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and proc...

Role Overview We are seeking an experienced GCP Data Engineer with strong expertise in designing, developing, and managing large-scale data pipelines on Google Cloud Platform. The ideal candidate will have hands-on experience in Big Data technologies (Hadoop, PySpark, BigQuery) and will collaborate closely with cross-functional teams and clients to deliver high-quality data solutions. Key Responsibilities Design, build, and maintain data pipelines and ETL workflows on GCP. Develop and optimize PySpark scripts and Spark-based data processing solutions. Work with BigQuery to manage datasets, write complex SQL queries, and optimize data performance. Utilize the Hadoop ecosystem (HDFS, Hive, Map...

Job Title: Databricks Developer Job Description:A Pyspark and Databricks Developer with a good understanding of the entire ETL/Azure lifecycle with a background of data projects. Responsibilities Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Data Factory, and other Azure services Implement and optimize Spark jobs, data transformations, and data processing workflows, Managing Databricks notebooks, Delta lake with Python, Delta Lake with Sparks SQL in Databricks Leverage Azure DevOps and CI/CD best practices to automate the deployment /DAB Deployments and management of data pipelines and infrastructure Ensure Data Integrity checks and Data Qual...

Job Description Experience in PySpark, Hive, and HBase. Proficiency in integrating, transforming, and consolidating data from various structured and unstructured data sources. Good experience in SQL or native SQL query languages. Good experience in PySpark, Hive, HBase, and Hadoop. Exposure to basic issues in working within Big Data clusters. Desired Skills Big Data Hadoop Python Spark Apache Spark Qualifications Bachelor of Technology.

5-6 years of total experience in data engineering or big data development. 2-3 years hands-on experience with Databricks and Apache Spark. Proficient in AWS cloud services (S3, Glue, Lambda, EMR, Redshift, CloudWatch, IAM). Strong programming skills in PySpark, Python, and optionally Scala. Solid understanding of data lakes, lakehouses, and Delta Lake concepts. Experience in SQL development and performance tuning. Familiarity with Airflow, dbt, or similar orchestration tools is a plus. Experience in CI/CD tools like Jenkins, GitHub Actions, or CodePipeline. Knowledge of data security, governance, and compliance frameworks. Qualifications: Bachelor of Engineering

Key Technical Skills: AWS tools (Redshift, Glue, EMR, Athena, StepFunction, CloudFormation, CloudWatch) Programming languages: SQL, Python, Pyspark Data Pipelining skills Data blending Big data technologies: Hadoop, Hive Data Ingestion and Integration: Ingest data from various sources such as databases, files, APIs, and streaming platforms into Azure. For example, use Azure Data Factory to orchestrate the extraction, transformation, and loading (ETL) of data from an on-premises SQL Server database to Azure Blob Storage. Data Transformation and Processing: Perform data transformation and processing tasks to prepare the data for analysis and reporting. Azure Databricks or Azure Synapse Analyti...

Hands-on development experience in programming languages such as Java and Scala using Maven, Apache Spark Frameworks , and Unix Shell scripting. Comfortable with Unix File System as well as HDFS commands. Experience with query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, and HBase DB. Should be flexible. Good communication and customer management skills. Desired Candidate Profile Qualifications : Undergraduate