Sr. Data Engineer

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. - Design and build efficient data pipelines using Azure Databricks (PySpark) - Implement business logic for data transformation and enrichment at scale - Manage and optimize Delta Lake storage solutions - Develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access Collaboration is essential in this role, as you will work cross-functionally with teams to deliver reliable, high-performance data solutions. It is important to follow best practices in code quality, version control, and documentation to ensure the success of data projects. Qualifications required for this position include: - At least 5 years of hands-on experience as a Data Engineer - Strong expertise in Azure Cloud services - Proficiency in Azure Databricks, PySpark, and Delta Lake - Solid experience in Python and FastAPI for API development - Experience with Azure Functions for serverless API deployments - Managing ETL pipelines using Apache Airflow - Working with PostgreSQL and MongoDB - Strong SQL skills and experience handling large datasets Preferred qualifications for this role include: - Familiarity with data ingestion from APIs or third-party data providers - Experience optimizing data pipelines for performance and scalability - Working knowledge of Azure services like Azure Data Lake and Azure Storage,