Azure Databricks Architect

Job Summary

SMBC Risk Technology Group is seeking a talented and motivated senior Azure Data Engineer. The role will be responsible for developing solutions for risk, regulatory, and compliance projects on Azure platform as well as on-prem to cloud migration initiatives. The Azure Cloud Engineer strategizes, designs, develops, and works with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions. The person will play a key role in supporting the growth and scale of Azure cloud-based technology infrastructure. This role will also be responsible for development of complex interfaces, troubleshooting issues, supporting weekend maintenance, and production support.

Primary Responsibilities

• Responsible for Administration, Configuration, and Optimization of Azure Databricks platform.
• Provide technical expertise to deploy applications on the platform.
• Work with data engineers, data scientists, and other stakeholders to ensure the platform meets requirements.
• Manage infrastructure and collaborate with teams to integrate with all components, including network, security, and remote access.
• Coordinate with SMBC Platform teams, APIM, Azure DevOps, and Support.
• Configure Databricks Cluster, and Unity Catalog repository.
• Integrate seamlessly with On-prem databases and Power BI reports.
• Build large-scale batch and real-time data pipelines with data processing frameworks in Azure cloud platform.
• Partner with architects, engineers, information analysts, business, and technology stakeholders.
• Develop and deploy enterprise grade platforms that enable data-driven solutions.
• Provide technical guidance on technology implementation and architectural changes.
• Analyze and design technical solutions on Azure to address business needs.
• Develop, test, and modify software to improve efficiency of data platforms and applications.
• Proficient in Python coding and frameworks knowledge.
• Own projects end to end, work with business stakeholders, IT teams, data stewards, and 3rd parties to ensure accurate and insightful outcomes.
• Integrate data from a variety of sources, ensure they adhere to data quality and accessibility standards.
• Enable streaming data use-cases and standard design patterns in a big data environment.
• Collaborate with data scientists to integrate algorithms and models into automated processes.