Job
Description
Seeking a Data Engineer with expertise in PySpark, Databricks, and high-throughput data lake architectures. The ideal candidate should have experience in workflow automation, data governance, and CDC implementation, with a deep understanding of data lineage and quality. Roles and Responsibilities: Design Build High-Throughput Data Lakes: Architect, develop, and optimize scalable and high-performance data lakes using PySpark and Databricks . Workflow Automation Data Pipelines: Implement and manage Databricks Workflows, Autoloaders, and Delta Live Tables (DLTs) for seamless data processing and ingestion. Optimize Data Storage Processing: Work with Parquet, Iceberg, and Hudi file formats, ensuring efficient partitioning, snapshotting, and compression strategies. Data Governance Quality: Establish data lineage, data governance, and data quality frameworks using Unity Catalog and industry best practices. Change Data Capture (CDC) Implementation: Design and implement CDC solutions using Debezium, AWS DMS, and other relevant tools. Cloud Infrastructure Security: Collaborate with cloud teams to deploy, monitor, and secure data infrastructure on AWS/GCP/Azure . Performance Optimization Monitoring: Continuously optimize data pipelines, queries, and jobs for improved efficiency and cost-effectiveness. Collaboration Knowledge Sharing: Work closely with engineering, analytics, and DevOps teams to ensure smooth data operations and knowledge transfer. Compliance Best Practices: Ensure data security, compliance, and privacy are maintained across all data workflows.
