Job Description

Key Responsibilities: Design, develop, and maintain scalable data pipelines using Scala/Python (PySpark) . Work with big data technologies to process and manage large datasets efficiently. Develop and optimize ETL workflows using Apache Airflow . Implement real-time and batch data processing solutions using Kafka and other streaming technologies. Build and maintain distributed databases (SQL & NoSQL) for efficient data storage and retrieval. Develop and manage CI/CD pipelines for data engineering workflows. Work with Google Cloud Platform (GCP) Data Tech Stack , including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage . Implement data governance, monitoring, and security best practices. Collaborate with data scientists, analysts, and software engineers to provide reliable data solutions. Optimize query performance and troubleshoot issues related to data pipelines and infrastructure. Required Skills & Qualifications: Strong programming skills in Scala and/or Python (PySpark) . Hands-on experience with big data technologies such as Hadoop, Spark, or Databricks. Experience in scripting (Shell/Python) for automation. Good understanding of GCP Data Tech Stack (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage). Experience with Apache Airflow for orchestrating data pipelines. Familiarity with CI/CD pipelines and DevOps practices. Experience with Kafka for real-time data streaming. Scala, Kafka, Aws, Gcp