Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Home
>
Jobs in bengaluru
>
UST
>
Associate III - Data Engineering

Associate III - Data Engineering

UST

3 - 5 years

5 Lacs

bengaluru

Posted:17 hours ago| Platform:

Apply

Skills Required

data glue data warehousing pyspark data pipeline sql analytics etl tool scd gcp awsazure bigquery etl programming communication skills python development data manipulation query airflow talend microsoft azure warehouse dataproc nosql data bricks aws glue concepts aws informatica

Work Mode

Work from Office

Job Type

Full Time

Job Description

Role Proficiency:

This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc with coding skills in Python PySpark and SQL. Works independently and demonstrates proficiency in at least one domain related to data with a solid understanding of SCD concepts and data warehousing principles.

Outcomes:

Collaborate closely with data analysts data scientists and other stakeholders to ensure data accessibility quality and security across various data sources.rnDesign develop and maintain data pipelines that collect process and transform large volumes of data from various sources.
Implement ETL (Extract Transform Load) processes to facilitate efficient data movement and transformation.
Integrate data from multiple sources including databases APIs cloud services and third-party data providers.
Establish data quality checks and validation procedures to ensure data accuracy completeness and consistency.
Develop and manage data storage solutions including relational databases NoSQL databases and data lakes.
Stay updated on the latest trends and best practices in data engineering cloud technologies and big data tools.

Measures of Outcomes:

Adherence to engineering processes and standards
Adherence to schedule / timelines
Adhere to SLAs where applicable
# of defects post delivery
# of non-compliance issues
Reduction of reoccurrence of known defects
Quickly turnaround production bugs
Completion of applicable technical/domain certifications
Completion of all mandatory training requirementst
Efficiency improvements in data pipelines (e.g. reduced resource consumption faster run times).
Average time to detect respond to and resolve pipeline failures or data issues.

Outputs Expected:

Code Development:

Develop data processing code independently
ensuring it meets performance and scalability requirements.

Documentation:

Create documentation for personal work and review deliverable documents
including source-target mappingstest casesand results.

Configuration:

Follow configuration processes diligently.

Testing:

Create and conduct unit tests for data pipelines and transformations to ensure data quality and correctness.
Validate the accuracy and performance of data processes.

Domain Relevance:

Develop features and components with a solid understanding of the business problems being addressed for the client.
Understand data schemas in relation to domain-specific contexts
such as EDI formats.

Defect Management:

Raise
fixand retest defects in accordance with project standards.

Estimation:

Estimate time
effortand resource dependencies for personal work.

Knowledge Management:

Consume and contribute to project-related documents
SharePointlibrariesand client universities.

Design Understanding:

Understand design and low-level design (LLD) and link it to requirements and user stories.

Certifications:

Obtain relevant technology certifications to enhance skills and knowledge.

Skill Examples:

Proficiency in SQL Python or other programming languages utilized for data manipulation.
Experience with ETL tools such as Apache Airflow Talend Informatica AWS Glue Dataproc and Azure ADF.
Hands-on experience with cloud platforms like AWS Azure or Google Cloud particularly with data-related services (e.g. AWS Glue BigQuery).
Conduct tests on data pipelines and evaluate results against data quality and performance specifications.
Experience in performance tuning data processes.
Proficiency in querying data warehouses.

Knowledge Examples:

Knowledge Examples

Knowledge of various ETL services provided by cloud providers including Apache PySpark AWS Glue GCP DataProc/DataFlow and Azure ADF/ADLF.
Understanding of data warehousing principles and practices.
Proficiency in SQL for analytics including windowing functions.
Familiarity with data schemas and models.
Understanding of domain-related data and its implications.

Additional Comments:

Role Scope / Deliverables: Create new data pipelines in databricks. Support existing data pipelines in databricks. Create DAG setup in in airflow. Resolve day to day job failures/performance issues. Key Skills: Minimum 2 years of total experience. Proficient in Databricks, Pyspark/Python & SQL (minimum 2 year of experience) Hands on experience in could AWS/Azure (preferably AWS) Excellent communication. Can be able to work independently.

Required Skills

Databricks,Pyspark,SQL

More Jobs at UST

Architect II - Technical SAP ABAP

Bengaluru

12 - 15 yrs

INR 0 - 0 Lacs

Lead I - SAP FICO with PS (Project Systems) and RAR

Bengaluru

5 - 7 yrs

INR 0 - 0 Lacs

Lead II - SAP ABAP

Bengaluru

7 - 9 yrs

INR 0 - 0 Lacs

Procurement

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Manual & Python Automation

Bengaluru, Karnataka

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now

UST

IT Services and IT Consulting

Aliso Viejo CA

Login to

Please Verify Your Phone or Email

Confirm Action

Associate III - Data Engineering