Senior Data Engineer

Job Description:

We are seeking a highly skilled and experienced Senior Data Engineer with a strong background in Microsoft Azure data platforms. The ideal candidate will have a proven track record of designing, implementing, and managing large-scale data solutions using Azure technologies. This role requires a deep understanding of data engineering best practices, cloud architecture, and the ability to work collaboratively with cross-functional teams.

The Sr. Data Engineer will design, implement, and maintain data integration projects focused primarily on the use of Azure SQL, Azure Databricks, Data Factory, Logic Apps, APIs, and Blob storage. This individual will work with business and technical staff to understand integration requirements and develop agile user stories to resolve them. They will also resolve customer complaints and respond to suggestions for improvements and enhancements. They will act as team leader for all data integration projects; coordinating the work of co-located and distributed team members, including vendor resources.

Essential Duties and Responsibilities:

Lead the design, implementation, testing, and maintenance of complex system integration processes focused primarily on the use of Azure SQL, Azure Databricks, Data Factory, Logic Apps, APIs, and Blob storage.

Coordinate the work of other systems integration technical resources, both onshore and offshore

Become an expert on Microsoft data integration and reporting products

Participate in requirement analysis and definition

Create technical specifications and system integration test plans

Document data flows using data dictionaries and other data governance tools

Collaborate with different technical teams on best practices

Work as part of an Agile development team to solve problems and develop projects in a fast-paced environment

Follow pre-established guidelines as well as create new guidelines to perform the functions of the job

Demonstrate a high degree of creativity and problem-solving skills

Mature the software development life cycle within your area of responsibility

Support applications using software development methodologies including structured programming, documentation, design, and code review and ensure consistent use of Azure DevOps to manage the full lifecycle

Collaborate with database resources to ensure robust and complete data access and manipulation

Collaborate with IT Operations to ensure hardware and software are aligned to deliver business requirements

Mentor junior and other new developers

Stay up to date with the latest Azure data services and best practices to continuously improve our data infrastructure.

Education, Experience and Knowledge:

Bachelor's / master's degree in computer science, engineering, or a related field, or in-lieu of degree, equivalent education, training, and work-related experience

10+ years of experience in IT with an emphasis on ETL, Reporting, and systems integration.

Strong expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure SQL Database, Azure Cosmos DB, Azure Data Lake, and Azure Databricks.

Proficiency in programming languages such as Python, SQL, Spark and Scala.

Experience with data modeling, ETL processes, and data warehousing / data lakes.

Experience designing and building relational databases

Expert knowledge of ETL concepts and techniques.

Proficient in SQL, stored procedures, triggers, indexes.

Experience integrating with SOAP and Restful Web Services

Experience applying enterprise design and architectural patterns

Experience in end-to-end Software Development Life Cycle (SDLC) projects

Experience with code review and evaluation

Good at understanding requirements and estimation

Team player, have ability and willingness to mentor junior developers

Have passion for learning new technologies and enhancing existing skills

Excellent problem-solving skills.

Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

Self-starter with ability to work independently.

Excellent Team player

Proactive issue resolution with a positive attitude

Ability to adapt and respond in a rapidly evolving business environment

Understands solution at the project level

Proficient in Microsoft Office (Word, Excel, Outlook, PowerPoint)

Preferred

Possess experience in the insurance industry and/ or basic knowledge of insurance or financial services.

Experience with Azure DevOps, CI/CD Pipelines and GIT source control

Experience with Azure Data Factory, Databricks, Azure SQL, API Management, Power BI

Experience with Azure virtual networks and private endpoints

Experience working with Agile/Scrum methodologies

Experience working with Denodo or any other Data Virtualization Tools

Exposure to Data science, AI/ML