Sr. Data Engineer

As a Sr. Data Engineer at NTT DATA in Bangalore, Karnataka (IN-KA), India, your primary responsibility is to build and implement PySpark-based data pipelines in Azure Synapse to transform and load data into ADLS in Delta format. You will design and implement dimensional (star/snowflake) and 3NF data models optimized for access using Power BI. Additionally, you will perform unit testing of data pipelines and transformations, design metadata-driven data pipelines using PySpark in Azure Synapse, and analyze and optimize Spark SQL queries. Collaborating with cross-functional teams to ensure data models align with business needs and maintaining version control and CI/CD pipelines in Git and Azure DevOps will also be part of your role. Key Responsibilities: - Build and implement PySpark-based data pipelines in Azure Synapse - Design and implement dimensional and 3NF data models optimized for Power BI access - Perform unit testing of data pipelines and transformations - Design metadata-driven data pipelines using PySpark in Azure Synapse - Analyze and optimize Spark SQL queries - Collaborate with cross-functional teams to align data models with business needs - Maintain version control and CI/CD pipelines in Git and Azure DevOps - Integrate Azure Purview for access controls and implement row level security - Perform Source-to-Target Mapping (STM) from source to multiple layers in the data lake Qualifications Required: - 7+ years of experience in SQL and PySpark - Hands-on experience with Azure Synapse, ADLS, Delta format, and metadata-driven data pipelines - Experience in implementing dimensional and 3NF data models - Proficiency in PySpark and Spark SQL, including query optimization and performance tuning - Strong problem-solving and analytical skills for debugging and optimizing data pipelines - Familiarity with CI/CD practices in Git and Azure DevOps - Working experience in an Azure DevOps-based development environment,