Senior Data Engineer

Role : Senior Data Engineer

Experience : 8-10 years experience

Skill Set Required :

Azure Data Factory

Azure Databricks

Python

SQL, PL/SQL

SSIS

Job Role/Job Description :

• Senior Data Engineer will be responsible for designing, implementing, and maintaining data solutions on the Microsoft Azure Data platform and SQL Server (SSIS, SSAS, UC4 Atomic)
• Collaborate with various stakeholders, and ensuring the efficient processing, storage, and retrieval of large volumes of data
• Technical Expertise and Responsibilities
• Design, build, and maintain scalable and reliable data pipelines.
• Should be able to design and build solutions in Azure data factory and Databricks to extract, transform and load data into different source and target systems.
• Should be able to design and build solutions in SSIS
• Should be able to analyze and understand the existing data landscape and provide recommendations/innovative ideas for rearchitecting / optimizing / streamlining to bring efficiency and scalability.
• Must be able to collaborate and effectively communicate with onshore counterparts to address technical gaps, requirement challenges, and other complex scenarios.
• Monitor and troubleshoot data systems to ensure high performance and reliability.
• Should be highly analytical and detail-oriented with extensive familiarity with database management principles.
• Optimize data processes for speed and efficiency.
• Ensure the data architecture supports business requirements and data governance policies.
• Define and execute the data engineering strategy in alignment with the companys goals.
• Integrate data from various sources, ensuring data quality and consistency.
• Stay updated with emerging technologies and industry trends.
• Understand the big picture business process utilizing deep knowledge in banking industry and translate them to data requirements.
• Enabling and running data migrations across different databases and different servers
• Perform thorough testing and validation to support the accuracy of data transformations and data verification used in machine learning models.
• Analyze data and different systems to define data requirements.
• Should be well versed with Data Structures & algorithms.
• Define data mapping working along with business and digital team and data team.
• Data pipeline maintenance/testing/performance validation
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Analyze and identify gaps on data needs and work with business and IT to bring in alignment on data needs.
• Troubleshoot and resolve technical issues as they arise.
• Optimize data flow and collection for cross-functional teams.
• Work closely with Data counterparts at onshore, product owners, and business stakeholders to understand data needs and strategies.
• Collaborate with IT and DevOps teams to ensure data infrastructure aligns with overall IT architecture.
• Implement best practices for data security and privacy.
• Drive continuous improvement initiatives within the data engineering function
• Optimize data flow and collection for cross-functional teams.
• Understand impact of data conversions as they pertain to servicing operations.
• Manage higher volume and more complex cases with accuracy and efficiency.

What is the expectation from the candidates current role/profile

• Design and develop warehouse solutions using Azure Synapse Analytics, ADLS, ADF, Databricks, Power BI, Azure Analysis Services
• Should be proficient in SSIS, SQL and Query optimization.
• Should have worked in onshore offshore model managing challenging scenarios.
• Expertise in working with large amounts of data (structured and unstructured), building data pipelines for ETL workloads and generate insights utilizing Data Science, Analytics.
• Expertise in Azure, AWS cloud services, and DevOps/CI/CD frameworks.
• Ability to work with ambiguity and vague requirements and transform them into deliverables.
• Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently.
• Drive automation efforts across the data analytics team utilizing Infrastructure as Code (IaC) using Terraform, Configuration Management, and Continuous Integration (CI) / Continuous Delivery (CD) tools such as Jenkins.
• Help build define architecture frameworks, best practices & processes. Collaborate on Data warehouse architecture and technical design discussions.
• Expertise in Azure Data factory and should be familiar with building pipelines for ETL projects.
• Expertise in SQL knowledge and experience working with relational databases.
• Expertise in Python and ETL projects
• Experience in data bricks will be of added advantage.
• Should have expertise in data life cycle, data ingestion, transformation, data loading, validation, and performance tuning.