Senior Infrastructure Test & Validation Engineer

Role Overview: You are being hired as a Senior Infrastructure Test & Validation Engineer with at least 10 years of experience to take charge of the Zero-Touch Validation, Upgrade, and Certification automation for the on-prem GPU cloud platform. Your main responsibility will be to ensure the stability, performance, and conformance of the entire stack, from hardware to Kubernetes, through automated, GitOps-based validation pipelines. Your expertise in infrastructure, along with hands-on skills in Sonobuoy, LitmusChaos, k6, and pytest, will be crucial for successful test orchestration, platform resilience, and continuous conformance. Key Responsibilities: - Design and implement automated, GitOps-compliant pipelines for validation and certification of the GPU cloud stack, covering hardware, OS, Kubernetes, and platform layers. - Integrate Sonobuoy for Kubernetes conformance and certification testing. - Design chaos engineering workflows using LitmusChaos to validate system resilience during failure scenarios. - Create performance testing suites with k6 and system-level benchmarks integrated into CI/CD pipelines. - Develop and maintain end-to-end test frameworks using pytest and/or Go, focusing on cluster lifecycle events, upgrade paths, and GPU workloads. - Ensure comprehensive test coverage and validation across conformance, performance, fault injection, and post-upgrade validation dimensions. - Establish dashboards and reporting mechanisms for automated test results, including traceability, drift detection, and compliance tracking. - Collaborate with infrastructure, SRE, and platform teams to incorporate testing and validation early in the deployment lifecycle. - Take ownership of quality assurance gates for all automation-driven deployments. Qualification Required: - At least 10 years of hands-on experience in infrastructure engineering, systems validation, or SRE roles. - Proficiency in pytest, Go, k6 scripting, and automation frameworks integration (Sonobuoy, LitmusChaos), along with CI integration. - Deep knowledge of Kubernetes architecture, upgrade patterns, and operational risks. - Experience in validating infrastructure components across lifecycle events. - Familiarity with GitOps workflows and CI/CD systems for test orchestration. - Strong scripting and automation skills in Python, Bash, or Go. - Understanding of GPU-based infrastructure and performance characteristics is advantageous. - Excellent debugging, root cause analysis, and incident investigation capabilities.,