Data Engineer - Data Ingestion

As a Data Engineer, your role will involve designing, building, and maintaining scalable data ingestion pipelines to onboard structured and semi-structured data from various sources. You will work with Azure-native services like Data Factory and Azure Data Lake, along with tools like Databricks or Apache Spark for data ingestion and transformation. Your responsibilities will include developing metadata-driven ingestion frameworks, collaborating with source system owners, and ensuring data quality, lineage, and governance principles are followed. Additionally, you will optimize ingestion processes for performance, reliability, and cloud cost efficiency, supporting both batch and real-time ingestion needs. Key Responsibilities: - Design, build, and maintain scalable and reusable data ingestion pipelines from APIs, flat files, databases, and external systems. - Work with Azure-native services and tools like Databricks or Apache Spark for data ingestion and transformation. - Develop metadata-driven ingestion frameworks for automated onboarding of new sources. - Collaborate with source system owners, analysts, and data stewards to define data ingestion specifications. - Implement monitoring and alerting on ingestion jobs to ensure data quality. - Optimize ingestion processes for performance, reliability, and cloud cost efficiency. - Support batch and real-time ingestion needs, including streaming data pipelines. Key Qualifications: - 3+ years of hands-on experience in data engineering, with a focus on data ingestion or integration. - Hands-on experience with Azure Data Services or equivalent cloud-native tools. - Experience in Python (PySpark) for data processing tasks. - Experience with ETL frameworks, orchestration tools, and API-based data ingestion. - Familiarity with data quality and validation strategies. - Understanding of CI/CD practices, version control, and infrastructure-as-code. - Bonus: Experience with streaming ingestion technologies like Kafka or Event Hubs. (Note: Any additional details about the company were not found in the provided job description.),