Analyse raw data from various source files (e.g. EXCEL CSV JSON XML Parquet etc.) to identify formatting issues inconsistencies missing values and discrepancies.

Use PySpark to transform clean and restructure source data to match the expected format for further processing and analysis.

Apply data validation rules and data quality checks to correct issues related to data integrity including handling null values duplicate records and data type mismatches.

Develop PySpark jobs to transform raw data into standardized formats (e.g. converting date formats normalizing text fields correcting encoding issues).

Ensure that data transformations meet the required schema and business logic by utilizing PySpark Data Frame and SQL functionalities.

Automate data transformation workflows and schedule data correction jobs to handle repetitive tasks efficiently.

Build interactive notebooks and workflows to perform data transformations and analytics within the Azure Data Bricks platform.

Develop and maintain data workflows in Azure Data Factory to orchestrate data movement and transformation across cloud-based storage and compute services.

Implement scheduled and event-driven data pipeline orchestration using ADF with focus on data quality performance and scalability.

Additional Details

Global Grade : C
Remote work possibility : Yes
Global Role Family : 60236 (P) Software Engineering
Local Role Name : 6504 Developer / Software Engineer
Local Skills : 35611 Azure Databricks
Languages Required: : English.

More Jobs at Growel Softech Pvt. Ltd.

Data Engineer

andhra pradesh

10.0 - 14.0 yrs

INR 8 - 16 Lacs

Tester

karnataka

7.0 - 10.0 yrs

INR 10 - 12 Lacs

DBA Engineer - 4b

maharashtra

3.0 - 5.0 yrs

INR 4 - 8 Lacs

SAP FI Professional

maharashtra

4.0 - 6.0 yrs

INR 5 - 8 Lacs

Pega Decisioning Professional

telangana, uttar pradesh

2.0 - 5.0 yrs

INR 4 - 7 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now

Growel Softech Pvt. Ltd.

Information Technology Services

Thane

Login to

Please Verify Your Phone or Email

Confirm Action

Data Engineer