Data Ingestion Engineer

As an Ingestion Engineer at Saxon Global, you will be responsible for designing, developing, and optimizing data ingestion pipelines to integrate multiple sources into Databricks. Your expertise in CI/CD and Kubernetes will be crucial in implementing and maintaining efficient data workflows. Collaboration with Data Engineers and stakeholders is essential to streamline data ingestion strategies and ensure data integrity, security, and compliance throughout the process. Key Responsibilities: - Design, develop, and optimize data ingestion pipelines for integrating multiple sources into Databricks. - Implement and maintain CI/CD pipelines for data workflows. - Deploy and manage containerized applications using Kubernetes. - Collaborate with Data Engineers and stakeholders to streamline data ingestion strategies. - Troubleshoot and optimize ingestion pipelines for performance and scalability. Required Skills & Qualifications: - Proven experience in data ingestion and pipeline development. - Hands-on experience with CI/CD tools such as GitHub Actions, Jenkins, Azure DevOps, etc. - Strong knowledge of Kubernetes and container orchestration. - Experience with Databricks, Spark, and data lake architectures. - Proficiency in Python, Scala, or SQL for data processing. - Familiarity with cloud platforms like AWS, Azure, or GCP. - Strong problem-solving and analytical skills. Preferred Qualifications: - Experience with Infrastructure as Code tools like Terraform, Helm, etc. - Background in streaming data ingestion technologies such as Kafka, Kinesis, etc. - Knowledge of data governance and security best practices.,