Big Data Engineer

Position Summary

Role Value Proposition

Job Responsibilities

• Collect, store, process and analyze large datasets to build and implement extract, transfer, load (ETL) processes
• Develop metadata and configuration based reusable frameworks to reduce the development effort
• Develop quality code with integral performance optimizations in place right at the development stage.
• Collaborate with global team in driving the delivery of projects and recommend development and performance improvements.
• Extensive experience of various databases types and knowledge to leverage the right one for the need.
• Strong understanding of data tools and ability to leverage them to understand the data and generate insights
• Hands on experience in building/designing at-scale Data Lake, Data warehouses, data stores for analytics consumption On prem and Cloud (real time as well as batch use cases)
• Ability to interact with business analysts and functional analysts in getting the requirements and implementing the ETL solutions.

Education, Technical Skills & Other Critical Requirement

Education

Experience

Technical Skills

1. Experience in transforming data using Python, Spark or Scala

2. Technical depth in Cloud Architecture Framework, Lakehouse Architecture and One Lake solutions.

3. Experience in implementing data ingestion and curation process using Azure with tools such as

   Azure Data Factory, Databricks Workflows, Azure Synapse, Cosmos DB, Spark (Scala/python), Data bricks

   .
4. Experience in cloud optimized code on Azure using Databricks,

   Synapse dedicated SQL Pool and serverless Pools,

   Cosmos, SQL APIs loading and consumption optimizations.
5. Scripting experience primarily on shell/bash/PowerShell would be desirable.
6. Experience in writing SQL and performing data analysis skills for data anomaly detection and data quality assurance.

Other Preferred Skills

1. Expertise in Python and experience writing Azure functions using Python/Node.js
2. Experience using

   Event Hub for data integrations

   .
3. Required working knowledge of Azure DevOps pipelines
4. Self-starter and ability to adapt with changing business needs