Jobs
Interviews
UST logo

Data Engineer

0 years

0 Lacs

Hyderabad Telangana India

Posted:1 month ago| Platform: Linkedin logo

Apply

Skills Required

data azure storage engineering pipeline test etl databricks analytics sql optimization scalability tuning security encryption compliance governance audit collaboration architecture design code documentation testing debugging coding resolve efficiency certifications python pyspark informatica vault aws gcp troubleshooting communication estimate ml ai integration metrics apache airflow scala

Work Mode

On-site

Job Type

Full Time

Job Description

Role:

Location:


Role Overview:

The Azure Big Data Engineer is a hands-on technical role focused on designing, implementing, and maintaining robust, scalable data pipelines and storage solutions on the Azure platform. The engineer plays a critical role in enabling data-driven decision-making across the organization by building and managing high-performance data infrastructure.


Key Responsibilities:

Data Engineering & Pipeline Development

  • Develop, test, and maintain ETL/ELT pipelines using tools such as Azure Data Factory, Azure Databricks, Synapse Analytics, and Microsoft Fabric.
  • Ingest, wrangle, transform, and join data from various sources ensuring data quality and consistency.
  • Implement data storage solutions using Azure Data Lake Storage, Lakehouse, Delta Lake, and data warehousing technologies (e.g., Synapse, Azure SQL Database).


Performance Optimization & Monitoring

  • Optimize data pipelines for cost, performance, scalability, and reliability.
  • Monitor data flows and proactively troubleshoot pipeline and performance issues.
  • Apply best practices for performance tuning and infrastructure alignment.


Security & Compliance

  • Implement data security measures including RBAC, data encryption, and compliance with organizational policies and regulatory standards.
  • Address data governance requirements and audit readiness.


Collaboration & Stakeholder Communication

  • Work closely with data scientists, architects, and analysts to gather requirements and deliver solutions aligned with business needs.
  • Translate business requirements into technical designs and documentation.
  • Present architecture and design options to stakeholders and conduct technical demos.


Expected Outcomes:

Delivery & Execution

  • Code pipelines and solutions following best practices in scalability, performance, and maintainability.
  • Complete documentation for architecture, source-target mappings, test cases, and performance benchmarks.
  • Perform unit testing, debugging, and performance validation of data pipelines.


Quality & Process Adherence

  • Ensure adherence to coding standards, project timelines, SLAs, and compliance protocols.
  • Quickly resolve production bugs and reduce recurring issues through RCA.
  • Improve pipeline efficiency (e.g., faster run times, reduced costs).


Knowledge & Certifications

  • Maintain up-to-date technical certifications and training.
  • Contribute to reusable documentation, knowledge bases, and process improvements.


Skills & Expertise:

Technical Skills

  • Programming:

    Proficiency in SQL, Python, and PySpark; familiarity with Scala.

  • ETL Tools:

    Azure Data Factory, Azure Databricks, Microsoft Fabric, Synapse Analytics, Informatica, Glue, DataProc.

  • Cloud Platforms:

    Expertise in Azure (including ADLS, Key Vault, Azure SQL, Synapse), familiarity with AWS or GCP a plus.

  • Data Warehousing:

    Snowflake, BigQuery, Delta Lake, Lakehouse.

  • Data Modeling:

    Strong understanding of dimensional modeling, schema design, and optimization for large datasets.

  • Security:

    Knowledge of data security and compliance best practices.


Soft Skills

  • Strong analytical and troubleshooting skills.
  • Excellent communication and collaboration capabilities.
  • Ability to estimate and manage workload effectively.
  • Customer-oriented approach with a focus on value delivery.


Preferred Qualifications:

  • Experience with Microsoft Fabric and modern data lakehouse architectures.
  • Exposure to ML/AI concepts and integration with data pipelines.
  • Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect).


Performance Metrics (KPIs):

  • Adherence to engineering standards and timelines.
  • Pipeline performance (run time, resource usage).
  • Reduction in post-release defects and production incidents.
  • Time to resolution for pipeline issues.
  • Completion of certifications and mandatory training.
  • Number of reusable assets created/shared.
  • Compliance with data governance and security policies.


Certifications:

  • Azure Data Engineer Associate (preferred)
  • Relevant domain certifications in data engineering, cloud, or big data technologies


Tools & Technologies:

  • ETL & Orchestration:

    Azure Data Factory, Databricks, Apache Airflow, Glue, Talend

  • Cloud Services:

    Azure Synapse, ADLS, Azure SQL, Key Vault, Microsoft Fabric

  • Programming:

    Python, SQL, PySpark, Scala (optional)

  • Data Platforms:

    Snowflake, BigQuery, Delta Lake, Azure Lakehouse

More Jobs at UST

Architect II - Technical SAP ABAP

Bengaluru

12 - 15 yrs
|

INR 0 - 0 Lacs

Lead I - SAP FICO with PS (Project Systems) and RAR

Bengaluru

5 - 7 yrs
|

INR 0 - 0 Lacs

Lead II - SAP ABAP

Bengaluru

7 - 9 yrs
|

INR 0 - 0 Lacs

Procurement

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Manual & Python Automation

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
UST logo
UST

IT Services and IT Consulting

Aliso Viejo CA

RecommendedJobs for You

Uplers logo

Data Engineer

Uplers

Chennai, Tamil Nadu, India

Tata Consultancy Services logo

Data Engineer

Tata Consultancy Services

Chennai, Tamil Nadu, India

MSCI Inc. logo

Data Engineer

MSCI Inc.

Pune, Maharashtra, India

Uplers logo

Data Engineer

Uplers

Pune, Maharashtra, India

Uplers logo

Data Engineer

Uplers

Gurugram, Haryana, India

Uplers logo

Data Engineer

Uplers

Noida, Uttar Pradesh, India

Tata Consultancy Services logo

Data Engineer

Tata Consultancy Services

Bengaluru, Karnataka, India

PANMACH logo

Data Engineer

PANMACH

Indore, Madhya Pradesh, India

R Systems logo

Data Engineer

R Systems

Delhi, India

MyCareernet logo

Data Engineer

MyCareernet

Hyderabad, Telangana, India