Data Platform Engineer

0 years

0 Lacs

Delhi, India

Posted:1 day ago| Platform: Linkedin logo

Apply Now

Skills Required

data spark deployment apache kubernetes scalability reliability monitoring optimization resolve management storage efficiency security authentication authorization encryption backup strategies integrity documentation troubleshooting collaboration devops containerization docker architecture tuning processing analytics communication orchestration helm sql mllib azure certification code terraform scripting python automation

Work Mode

On-site

Job Type

Contractual

Job Description

Spark Cluster Deployment: Deploy, configure, and maintain Apache Spark clusters on Kubernetes, ensuring scalability, reliability, and performance. Application Deployment: Collaborate with data engineers and data scientists to deploy Spark applications and workloads, ensuring they run efficiently. Monitoring and Optimization: Implement monitoring solutions to track cluster performance, resource utilization, and application health. Proactively identify and resolve performance bottlenecks. Resource Management: Manage cluster resources, including CPU, memory, and storage allocation, to ensure optimal utilization and cost efficiency. Security: Implement and maintain security measures, including authentication, authorization, and encryption, to protect sensitive data and Spark clusters. Backup and Recovery: Develop and maintain backup and recovery strategies to ensure data integrity and availability in case of failures. Documentation: Maintain clear and comprehensive documentation of Spark cluster configurations, deployment procedures, and best practices. Troubleshooting: Quickly diagnose and resolve issues related to Spark clusters, applications, and Kubernetes infrastructure. Collaboration: Work closely with cross-functional teams, including data engineers, data scientists, and DevOps, to understand application requirements and optimize Spark clusters accordingly. Requirements Proven experience deploying and managing Apache Spark on Kubernetes in a production environment. Proficiency in containerization technologies, particularly Docker and Kubernetes. Strong knowledge of Spark architecture, including cluster, driver, and worker nodes. Familiarity with Spark tuning, optimization, and performance monitoring. Experience with resource management tools like Kubernetes Resource Quotas and LimitRanges. Understanding of data processing and analytics workflows. Excellent problem-solving and troubleshooting skills. Strong communication and collaboration skills. Experience with Spark cluster orchestration tools like Helm. Knowledge of Spark ecosystem components such as Spark SQL, Spark Streaming, and MLlib. Familiarity with cloud-based solution (Azure). Certification in Kubernetes (e.g., Certified Kubernetes Administrator - CKA). Knowledge of CI/CD pipelines and infrastructure as code (IaC) tools (e.g., Terraform). Scripting skills in languages like Python, Bash, or Shell. Understanding of DevOps practices and automation. Show more Show less

Mock Interview

Boost Confidence & Sharpen Skills

Start Data Interview Now

RecommendedJobs for You

Indore, Madhya Pradesh, India