Home
Jobs
Companies
Resume
UST logo

Associate III - Data Engineering

5 - 8 years

0 Lacs

Trivandrum, Kerala, India

Posted:1 month ago| Platform: Linkedin logo

Apply

Skills Required

data engineering pipeline development coding testing etl informatica databricks python pyspark sql accessibility security extract checks storage relational nosql schedule compliance training resolve code processing scalability documentation test configuration management estimation estimate sharepoint design certifications technology programming apache airflow talend aws azure tuning querying gcp dataflow adf analytics reliability architecture database communication collaboration devops

Work Mode

Not specified

Job Type

Full Time

Job Description

Role Description Role Proficiency: This role requires proficiency in data pipeline development including coding and testing data pipelines for ingesting wrangling transforming and joining data from various sources. Must be adept at using ETL tools such as Informatica Glue Databricks and DataProc with coding skills in Python PySpark and SQL. Works independently and demonstrates proficiency in at least one domain related to data with a solid understanding of SCD concepts and data warehousing principles. Outcomes Collaborate closely with data analysts data scientists and other stakeholders to ensure data accessibility quality and security across various data sources.rnDesign develop and maintain data pipelines that collect process and transform large volumes of data from various sources.Implement ETL (Extract Transform Load) processes to facilitate efficient data movement and transformation.Integrate data from multiple sources including databases APIs cloud services and third-party data providers.Establish data quality checks and validation procedures to ensure data accuracy completeness and consistency.Develop and manage data storage solutions including relational databases NoSQL databases and data lakes.Stay updated on the latest trends and best practices in data engineering cloud technologies and big data tools. Measures Of Outcomes Adherence to engineering processes and standardsAdherence to schedule / timelinesAdhere to SLAs where applicable# of defects post delivery# of non-compliance issuesReduction of reoccurrence of known defectsQuickly turnaround production bugsCompletion of applicable technical/domain certificationsCompletion of all mandatory training requirementstEfficiency improvements in data pipelines (e.g. reduced resource consumption faster run times).Average time to detect respond to and resolve pipeline failures or data issues. Outputs Expected Code Development: Develop data processing code independently ensuring it meets performance and scalability requirements. Documentation Create documentation for personal work and review deliverable documents including source-target mappings test cases and results. Configuration Follow configuration processes diligently. Testing Create and conduct unit tests for data pipelines and transformations to ensure data quality and correctness.Validate the accuracy and performance of data processes. Domain Relevance Develop features and components with a solid understanding of the business problems being addressed for the client.Understand data schemas in relation to domain-specific contexts such as EDI formats. Defect Management Raise fix and retest defects in accordance with project standards. Estimation Estimate time effort and resource dependencies for personal work. Knowledge Management Consume and contribute to project-related documents SharePoint libraries and client universities. Design Understanding Understand design and low-level design (LLD) and link it to requirements and user stories. Certifications Obtain relevant technology certifications to enhance skills and knowledge. Skill Examples Proficiency in SQL Python or other programming languages utilized for data manipulation.Experience with ETL tools such as Apache Airflow Talend Informatica AWS Glue Dataproc and Azure ADF.Hands-on experience with cloud platforms like AWS Azure or Google Cloud particularly with data-related services (e.g. AWS Glue BigQuery).Conduct tests on data pipelines and evaluate results against data quality and performance specifications.Experience in performance tuning data processes.Proficiency in querying data warehouses. Knowledge Examples Knowledge Examples Knowledge of various ETL services provided by cloud providers including Apache PySpark AWS Glue GCP DataProc/DataFlow and Azure ADF/ADLF.Understanding of data warehousing principles and practices.Proficiency in SQL for analytics including windowing functions.Familiarity with data schemas and models.Understanding of domain-related data and its implications. Additional Comments Design, develop, and maintain data pipelines and architectures using Azure services. Collaborate with data scientists and analysts to meet data needs. Optimize data systems for performance and reliability. Monitor and troubleshoot data storage and processing issues. Responsibilities Design, develop, and maintain data pipelines and architectures using Azure services. Collaborate with data scientists and analysts to meet data needs. Optimize data systems for performance and reliability. Monitor and troubleshoot data storage and processing issues. Ensure data security and compliance with company policies. Document data solutions and architecture for future reference. Stay updated with Azure data engineering best practices and tools. Qualifications Bachelor's degree in Computer Science, Information Technology, or a related field. 3+ years of experience in data engineering. Proficiency in Azure Data Factory, Azure SQL Database, and Azure Databricks. Experience with data modeling and ETL processes. Strong understanding of database management and data warehousing concepts. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills. Skills Azure Data Factory Azure SQL Database Azure Databricks ETL Data Modeling SQL Python Big Data Technologies Data Warehousing Azure DevOps Skills Azure,Aws,Aws Cloud,Azure Cloud

More Jobs at UST

UCC Engineer (Collab)

Trivandrum

5 - 7 yrs
|

INR 0 - 0 Lacs

Specialist I - Cloud Infrastructure Services - Network Engineer

Trivandrum

12 - 18 yrs
|

INR 0 - 0 Lacs

Application Packaging - SCCM, Release management

Trivandrum

8 - 12 yrs
|

INR 0 - 0 Lacs

Lead I - Cloud Infrastructure Services

Trivandrum

5 - 7 yrs
|

INR 0 - 0 Lacs

SQL Database Engineering

Trivandrum

12 - 15 yrs
|

INR 0 - 0 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Data Interview Now
UST
UST
www.ust.com

IT Services and IT Consulting

Aliso Viejo CA

10001 Employees

1845 Jobs

    Key People

  • Kris Canekeratne

    Co-Founder & CEO

  • Sandeep Reddy

    President

RecommendedJobs for You

UST logo

Associate III - Data Engineering

UST

Chennai, Tamil Nadu, India

UST logo

Associate III - Data Engineering

UST

Trivandrum, Kerala, India

UST logo

Associate III - Data Engineering

UST

Trivandrum

UST logo

Associate III - Data Engineering

UST

Trivandrum, Kerala, India

UST logo

Associate III - Data Engineering

UST

Chennai, Tamil Nadu, India

UST logo

Associate III - Data Engineering

UST

Trivandrum, Kerala, India

UST logo

Associate III - Data Engineering

UST

Trivandrum, Kerala, India