Lead

As a Data Engineer at UST, your role involves designing, developing, and optimizing ETL pipelines using PySpark on Google Cloud Platform (GCP). You will work with various GCP services such as BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration. Your responsibilities will include: - Developing and optimizing Spark-based ETL processes for large-scale data processing. - Implementing best practices for data governance, security, and monitoring in a cloud environment. - Collaborating with data engineers, analysts, and business stakeholders to understand data requirements. - Troubleshooting performance bottlenecks and optimizing Spark jobs for efficient execution. - Automating data workflows using Apache Airflow or Cloud Composer. - Ensuring data quality, validation, and consistency across pipelines. Qualifications required for this role include: - 5+ years of experience in ETL development with a focus on PySpark. - Strong hands-on experience with Google Cloud Platform (GCP) services such as BigQuery, Cloud Dataflow, Cloud Composer, and Cloud Storage. - Proficiency in Python and PySpark for big data processing. - Experience with data lake architectures and data warehousing concepts. - Knowledge of SQL for data querying and transformation. - Experience with CI/CD pipelines for data pipeline automation. - Strong debugging and problem-solving skills. - Experience with Kafka or Pub/Sub for real-time data processing. - Knowledge of Terraform for infrastructure automation on GCP. - Experience with containerization using Docker and Kubernetes. - Familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog.,