Data Engineer

You are applying for the position of Data Engineer at Proplens, a client of Uplers. **Role Overview:** As a Data Engineer with 4-7 years of experience, you will be responsible for owning end-to-end data pipelines for downstream AI agent applications. Your role will involve designing data models, building scalable ETL/ELT workflows, and collaborating on ML model deployment within the AI agent space. **Key Responsibilities:** - Data Modeling & Pipeline Development - Define and implement logical/physical data models and schemas - Develop schema mapping and data dictionary artifacts for cross-system consistency - Build, test, and maintain robust ETL/ELT workflows using Spark (batch & streaming) - Automate data ingestion from diverse sources (Databases, APIs, files, Sharepoint/document management tools, URLs) - Gen AI Integration - Collaborate with AI engineers to enrich data for agentic workflows - Instrument pipelines to surface real-time context into LLM prompts - ML Model Deployment Support (Secondary role) - Package and deploy ML models (e.g., via MLflow, Docker, or Kubernetes) - Integrate inference endpoints into data pipelines for feature serving - Observability & Governance - Implement monitoring, alerting, and logging (data quality, latency, errors) - Apply access controls and data privacy safeguards (e.g., Unity Catalog, IAM) - CI/CD & Automation - Develop automated testing, versioning, and deployment (Azure DevOps, GitHub Actions, Airflow) - Maintain reproducible environments with infrastructure as code (Terraform, ARM templates) **Qualifications Required:** - 5 years in Data Engineering or similar role, with exposure to ML modeling pipelines - Proficiency in Python, dlt for ETL/ELT pipeline, duckDB for analytical in-process analysis, dvc for managing large files efficiently. - Solid SQL skills and experience designing and scaling relational databases. - Familiarity with non-relational column based databases. - Familiarity with Prefect is preferred or others (e.g. Azure Data Factory) - Proficiency with the Azure ecosystem. Should have worked on Azure services in production. - Familiarity with RAG indexing, chunking, and storage across file types for efficient retrieval. - Experience deploying ML artifacts using MLflow, Docker, or Kubernetes - Strong Dev Ops/Git workflows and CI/CD (CircleCI and Azure DevOps) **About the Company:** Proplens is an early-stage startup offering fast-growing, revenue-generating proptech solutions. The company provides steep learning opportunities in real-world enterprise production use-cases, offers remote work with quarterly meet-ups, and exposure to multi-market clients. If you believe you are suited for this fast-paced environment and are ready for a new challenge with real and direct impact in production, please apply through the portal following the steps mentioned. **Note:** Uplers" goal is to make hiring reliable, simple, and fast, by helping talents find and apply for relevant contractual onsite opportunities and progress in their careers. You will receive support for any grievances or challenges faced during the engagement.,