Genzeon - Databricks Migration Consultant/Architect

Databricks Migration Architect / Consultant 8+

• Conduct comprehensive cost analysis and auditing of existing Databricks deployments across multiple workspaces
• Develop and implement aggressive cost reduction strategies targeting 30-50% savings through cluster optimization
• Design and deploy automated cost monitoring solutions with real-time alerts and budget controls
• Optimize cluster configurations, auto-scaling policies, and job scheduling to minimize compute costs
• Implement spot instance strategies and preemptible VM usage for non-critical workloads
• Establish cost allocation frameworks and implement chargeback mechanisms for business unit accountability
• Create cost governance policies and developer guidelines to prevent cost overruns
• Analyze and optimize storage costs including Delta Lake table optimization and data lifecycle from Databricks Architecture :
• Lead strategic initiatives to migrate workloads away from Databricks to cost-effective alternatives
• Assess existing Databricks implementations and create detailed migration roadmaps to target platforms
• Design migration architectures for transitioning to open-source Spark on Kubernetes, EMR, or other platforms
• Develop automated migration tools and frameworks to minimize business disruption
• Create comprehensive migration strategies including data export, job conversion, and dependency mapping
• Establish parallel running environments to ensure zero-downtime migrations
• Lead post-migration validation and performance benchmarking against original Databricks solutions
• Document lessons learned and create reusable migration playbooks for future Spark Cluster Solutions :
• Design high-performance, cost-optimized Spark cluster architectures outside of Databricks ecosystem
• Implement custom Spark solutions on Kubernetes, YARN, and standalone cluster managers
• Optimize Spark job performance through advanced tuning of memory management, serialization, and parallelism
• Develop custom Spark operators and applications for specialized business use cases
• Troubleshoot complex Spark performance bottlenecks and implement optimization strategies
• Create cluster auto-scaling solutions and dynamic resource allocation frameworks
• Design fault-tolerant Spark architectures with disaster recovery and high availability
• Implement monitoring and alerting for Spark cluster health and job performance Planning & Execution :
• Collaborate with finance teams to develop multi-year cost reduction roadmaps
• Evaluate and recommend alternative platforms based on cost-benefit analysis
• Create business cases for migration projects with detailed ROI calculations
• Establish technical debt reduction strategies related to Databricks dependencies
• Partner with procurement teams on contract negotiations and vendor Qualifications : Experience :
• 8+ years of experience in big data architecture with focus on cost optimization
• 5+ years of hands-on Databricks experience with proven cost reduction achievements
• Demonstrated experience architecting and executing complete platform migrations from Databricks to alternative solutions with successful outcomes
• 6+ years of advanced Apache Spark development and cluster management experience
• Track record of achieving significant cost savings (minimum 40%+) in cloud data Optimization Expertise :
• Expert knowledge of Databricks pricing models, compute types, and cost drivers
• Experience with FinOps practices and cloud cost management tools
• Proven ability to implement automated cost controls and budget management systems
• Knowledge of alternative platforms and their cost structures (EMR, HDInsight, GCP Dataproc, & Spark Technical Skills :
• Deep expertise in migrating complex data workloads between different Spark platforms
• Advanced knowledge of Spark internals, catalyst optimizer, and performance tuning
• Experience with Kubernetes-based Spark deployments and container orchestration
• Proficiency in infrastructure-as-code for multi-cloud Spark cluster provisioning
• Strong background in data pipeline migration and ETL/ELT conversion & Platform Skills :
• Expert-level proficiency in Scala, Python, and Java for Spark development
• Advanced SQL skills and experience with multiple database technologies
• Experience with open-source alternatives to Databricks (Apache Spark, Delta Lake OSS, MLflow OSS)
• Knowledge of streaming platforms (Kafka, Kinesis, Pulsar) and real-time architectures
• Proficiency with monitoring tools (Prometheus, Grafana, ELK Qualifications :
• Databricks certifications combined with experience in competitive platforms
• Cloud cost management certifications (AWS Cost Optimization, Azure Cost Management)
• Experience with vendor negotiations and contract optimization
• Background in building business cases for platform migrations
• Knowledge of data governance during platform transitions
• Experience with Apache Iceberg, Hudi, or other open table formats as Delta Lake :
• Bachelor's degree in Computer Science, Engineering, Information Technology, or related field
• Master's degree preferred but not required with equivalent experience