Sr. Data Engineer

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. Key Responsibilities: - Design and build efficient data pipelines using Azure Databricks (PySpark) - Implement business logic for data transformation and enrichment at scale - Manage and optimize Delta Lake storage solutions - Develop REST APIs using FastAPI to expose processed data - Deploy REST APIs on Azure Functions for scalable and serverless data access - Develop and manage Airflow DAGs to orchestrate ETL processes - Ingest and process data from various internal and external sources on a scheduled basis - Manage database storage and access using PostgreSQL and MongoDB - Write optimized SQL queries to support downstream applications and analytics - Follow best practices in code quality, version control, and documentation Qualifications Required: - At least 5 years of hands-on experience as a Data Engineer - Strong expertise in Azure Cloud services - Proficiency in Azure Databricks, PySpark, and Delta Lake - Solid experience in Python and FastAPI for API development - Experience with Azure Functions for serverless API deployments - Managing ETL pipelines using Apache Airflow - Working with PostgreSQL and MongoDB - Strong SQL skills and experience handling large datasets Preferred qualifications include familiarity with data ingestion from APIs or third-party data providers, experience optimizing data pipelines for performance and scalability, and a working knowledge of Azure services like Azure Data Lake and Azure Storage.,