Data Engineer - Data Ingestion

As a Data Engineer at our company, your role will be crucial in designing, building, and maintaining scalable data ingestion pipelines to ensure the delivery of high-quality, reliable, and scalable data pipelines across our Data & AI ecosystem. Your main responsibilities will include: - Designing, building, and maintaining scalable and reusable data ingestion pipelines to onboard structured and semi-structured data from APIs, flat files, databases, and external systems. - Working with Azure-native services such as Data Factory, Azure Data Lake, Event Hubs, and tools like Databricks or Apache Spark for data ingestion and transformation. - Developing and managing metadata-driven ingestion frameworks to support dynamic and automated onboarding of new sources. - Collaborating closely with source system owners, analysts, and data stewards to define data ingestion specifications and implement monitoring/alerting on ingestion jobs. - Ensuring data quality, lineage, and governance principles are embedded into ingestion processes. - Optimizing ingestion processes for performance, reliability, and cloud cost efficiency. - Supporting batch and real-time ingestion needs, including streaming data pipelines where applicable. Key qualifications for this role include: - 3+ years of hands-on experience in data engineering with a specific focus on data ingestion or integration. - Hands-on experience with Azure Data Services (e.g., ADF, Databricks, Synapse, ADLS) or equivalent cloud-native tools. - Experience in Python (PySpark) for data processing tasks (bonus: SQL knowledge). - Experience with ETL frameworks, orchestration tools, and working with API-based data ingestion. - Familiarity with data quality and validation strategies, including schema enforcement and error handling. - Good understanding of CI/CD practices, version control, and infrastructure-as-code (e.g., Terraform, Git). - Bonus: Experience with streaming ingestion (e.g., Kafka, Event Hubs, Spark Structured Streaming).,