Spark Engineer

As a Spark Technical Solutions Engineer, your role involves providing deep dive technical and consulting solutions for challenging Spark/ML/AI/Delta/Streaming/Lakehouse issues reported by customers. Utilizing your comprehensive technical skills and effective communication, you will assist customers in their Data bricks journey, offering guidance and expertise to help them achieve their strategic objectives using the company's products. Key Responsibilities: - Perform initial level analysis and troubleshoot Spark issues using Spark UI metrics, DAG, and Event Logs - Troubleshoot and suggest code-level analysis of Spark to address customer issues related to Spark core internals, Spark SQL, Structured Streaming, Delta, Lakehouse, and other Data bricks runtime features - Assist customers in setting up reproducible Spark problems in areas such as Spark SQL, Delta, Memory Management, Performance tuning, Streaming, Data Science, and Data Integration - Participate in the Designated Solutions Engineer program to address strategic customer Spark and Cloud issues - Plan and coordinate with internal teams for customer issue resolution and best practices guidelines - Build and maintain internal wiki and knowledge base for support and customers - Coordinate with Engineering and Support teams to identify and report product defects - Provide on-call support during runtime outages and escalate critical customer operational issues - Offer best practices guidance on Spark runtime performance and usage of Spark core libraries and APIs - Advocate for customer needs and contribute to tools/automation initiatives - Support third party integrations with Data bricks environment - Manage assigned Spark cases daily and adhere to SLA commitments - Exceed support organization KPI expectations - Continuously enhance expertise in AWS/Azure and Data bricks platform through learning and training programs Qualifications Required: - Minimum 6 years of experience in designing, building, testing, and maintaining Python/Java/Scala applications - 3 years of hands-on experience in Big Data, Hadoop, Spark, Machine Learning, AI, Streaming, Kafka, Data Science, or Elasticsearch - Hands-on experience in performance tuning and troubleshooting of Hive and Spark applications - Real-time experience in JVM and Memory Management techniques preferred - Working experience with SQL-based databases, Data Warehousing/ETL technologies, and AWS/Azure/GCP preferred - Excellent written and oral communication skills - Linux/Unix administration skills are a plus - Knowledge in Data Lakes and experience with SCD use cases at production scale - Strong analytical and problem-solving skills in Distributed Big Data Computing environment (Note: The competencies section has been omitted as it does not directly relate to the job responsibilities and qualifications),