8 - 12 years
Posted:1 week ago
Job purpose
The Senior Databricks Platform & Data Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and realtime pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.
Key responsibilities
Key competencies
Technical Competencies
Databricks Platform Engineering: Advanced hands-on capability in administering, configuring, deploying, monitoring, tuning, scaling, and supporting Databricks on AWS.
Python, SQL and PySpark Engineering: Strong programming capability for developing, testing, optimizing, and troubleshooting production-grade pipelines.
Kafka and Structured Streaming: Expertise in real-time ingestion, event-driven processing, fault tolerance, checkpointing, schema evolution, latency management, and streaming recovery.
Unity Catalog, Security and Governance: Experience implementing data lineage, metadata governance, permissions, auditing, data isolation, secure sharing, and least-privilege access.
AWS Infrastructure and Networking: Working knowledge of IAM, VPCs, subnets, routing, security groups, private connectivity, S3, EC2, CloudWatch, KMS, Secrets Manager, and firewall dependencies.
Performance, Scalability and Cost Optimization: Ability to tune Spark jobs, SQL queries, streaming pipelines,
Delta tables, clusters, and serverless compute while managing cost and reliability.
Deployment and Automation: Experience with Git, CI/CD, Databricks Asset Bundles, automated testing, deployment automation, workflow orchestration, and operational monitoring.
Architecture and Trade-off Analysis: Ability to assess serverless, classic, and hybrid approaches based on security, cost, performance, scalability, supportability, and operational risk.
Behavioral Competencies
Production Ownership and Problem-Solving: Ability to independently diagnose and resolve complex data, platform, networking, security, streaming, and performance issues.
Cross-functional Collaboration and Communication: Ability to work effectively with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, firewall, and source-system teams.
Free courses, interview questions and practice for each skill.
Acuity knowledge Partners
Upload Resume
Drag or click to upload
Your data is secure with us, protected by advanced encryption.
Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.
We have sent an OTP to your contact. Please enter it below to verify.
Practice Python coding challenges to boost your skills
Start Practicing Python Now
Salary: Not disclosed
Salary: Not disclosed
AI Job Matching Summary