Job
Description
Job Description Job Location: Hyderabad Job Duration: Full time Hours: 9:00am to 5:00pm We are seeking a hands-on Data Engineer with a strong focus on data ingestion to support the delivery of high-quality, reliable, and scalable data pipelines across our Data & AI ecosystem. This role is essential in enabling downstream analytics, machine learning, and business intelligence solutions by ensuring robust and automated data acquisition from various internal and external sources. Key Responsibilities Design, build, and maintain scalable and reusable data ingestion pipelines to onboard structured and semi-structured data from APIs, flat files, databases, and external systems. Work with Azure-native services (e.g., Data Factory, Azure Data Lake, Event Hubs) and tools like Databricks or Apache Spark for data ingestion and transformation. Develop and manage metadata-driven ingestion frameworks to support dynamic and automated onboarding of new sources. Collaborate closely with source system owners, analysts, and data stewards to define data ingestion specifications and implement monitoring/alerting on ingestion jobs. Ensure data quality, lineage, and governance principles are embedded into ingestion processes. Optimize ingestion processes for performance, reliability, and cloud cost efficiency. Support batch and real-time ingestion needs, including streaming data pipelines where applicable. Technical Experience 3+ years of hands-on experience in data engineering – bonus: with a specific focus on data ingestion or integration. Hands-on experience with Azure Data Services (e.g., ADF, Databricks, Synapse, ADLS) or equivalent cloud-native tools. Experience in Python (PySpark) for data processing tasks. (bonus: SQL knowledge) Experience with ETL frameworks, orchestration tools, and working with API-based data ingestion. Familiarity with data quality and validation strategies, including schema enforcement and error handling. Good understanding of CI/CD practices, version control, and infrastructure-as-code (e.g., Terraform, Git). Bonus: Experience with streaming ingestion (e.g., Kafka, Event Hubs, Spark Structured Streaming).
