Spark Developer

About the Role:

Spark Developer

Key Responsibilities:

• Design, develop, and maintain

  data processing pipelines

  using

  Apache Spark (Core, SQL, Streaming, Structured Streaming)

  .
• Implement ETL workflows for data ingestion, transformation, and integration across multiple data sources.
  
• Optimize Spark applications for

  performance, scalability, and reliability

  in distributed environments.
• Work with large structured and unstructured datasets to support analytical and reporting needs.
  
• Collaborate with data scientists and business analysts to understand data requirements and provide engineering solutions.
  
• Integrate Spark applications with cloud platforms (AWS, Azure, GCP) and data storage systems (HDFS, S3, Hive, Cassandra, etc.).

Preferred Skills:

• Experience with

  cloud-based data platforms

  (AWS EMR, Databricks, Azure Synapse, GCP Dataproc).
• Familiarity with

  containerization (Docker, Kubernetes)

  and

  orchestration tools (Airflow, Luigi)

  .
• Knowledge of

  data lake architectures

  ,

  Delta Lake

  , or

  Parquet/Avro

  file formats.
• Exposure to

  machine learning pipelines

  and

  data science integrations

  .
• Understanding of

  data security, compliance, and governance principles

  .