Sr. Python Developer

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). Your role will involve implementing business logic for data transformation and enrichment at scale and managing and optimizing Delta Lake storage solutions. You will also be involved in developing REST APIs using FastAPI to expose processed data and deploying APIs on Azure Functions for scalable and serverless data access. Additionally, you will develop and manage Airflow DAGs to orchestrate ETL processes, ingest and process data from various internal and external sources on a scheduled basis. Database management will be a key part of your responsibilities, including handling data storage and access using PostgreSQL and MongoDB. You will write optimized SQL queries to support downstream applications and analytics. Collaboration is an essential aspect of this role, requiring you to work cross-functionally with teams to deliver reliable, high-performance data solutions. You will need to follow best practices in code quality, version control, and documentation. To excel in this role, you should have at least 5 years of hands-on experience as a Data Engineer. Strong experience with Azure Cloud services is required, along with proficiency in Azure Databricks, PySpark, and Delta Lake. Solid experience with Python and FastAPI for API development, as well as experience with Azure Functions for serverless API deployments, is essential. You should also be skilled in managing ETL pipelines using Apache Airflow and have hands-on experience with PostgreSQL and MongoDB. Strong SQL skills and experience handling large datasets are also necessary.,