Data Engineer

Primary Responsibilities:

• Azure Infrastructure Management: Own and maintain all aspects of Azure infrastructure, recommending modifications to enhance reliability, availability, and scalability
• Security Management: Manage security aspects of Azure infrastructure, including network, firewall, private endpoints, encryption, PIM, and permissions management using Azure RBAC and Databricks roles
• Technical Troubleshooting: Diagnose and troubleshoot technical issues in a timely manner, identifying root causes and providing effective solutions
• Infrastructure as Code: Create and maintain Azure Infrastructure as Code using Terraform and GitHub Actions
• CI/CD Pipelines: Configure and maintain CI/CD pipelines using GitHub Actions for various Azure services such as ADF, Databricks, Storage, and Key Vault
• Programming Expertise: Utilize your expertise in programming languages such as Python to develop and maintain data engineering solutions
• Generative AI and Language Models: Knowledge of Language Models (LLMs) and Generative AI is a plus, enabling the integration of advanced AI capabilities into data workflows
• Real-Time Data Streaming: Use Kafka for real-time data streaming and integration, ensuring efficient data flow and processing
• Data Management: Proficiency in Snowflake for data wrangling and management, optimizing data structures for analysis
• DBT Utilization: Build and maintain data marts and views using DBT, ensuring data is structured for optimal analysis
• ETL/ELT Solutions: Design ETL/ELT solutions using tools like Azure Data Factory and Azure Databricks, leveraging methodologies to acquire data from various structured or semi-structured source systems
• Communication: Solid communication skills to explain technical issues and solutions clearly to the Engineering Lead and key stakeholders (as required)
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so