Data Engineer - Python / PySpark

Job Summary :

solutions and pipelines.

Key Responsibilities :

- Build and maintain scalable ETL/ELT pipelines with a focus on performance and reliability.

- Write efficient and reusable code using Python and PySpark.

- Perform data cleansing, transformation, and migration across various platforms.

- Work hands-on with Azure Data Factory (ADF) for at least 1.5 to 2 years.

- Develop and optimize SQL queries, stored procedures, and manage large data sets using SQL Server, T-SQL, PL/SQL, etc.

- Collaborate with cross-functional teams to understand business requirements and provide data-driven solutions.

- Engage directly with clients and business stakeholders to gather requirements, suggest optimal solutions, and ensure successful delivery.

- Work with Power BI for basic reporting and data visualization tasks.

- Apply strong knowledge of data warehousing concepts, modern data platforms, and cloud-based analytics.

- Adhere to coding standards and best practices, including thorough documentation and testing (unit, integration, performance).

- Support the operations, maintenance, and enhancement of existing data pipelines and architecture.

- Estimate tasks and plan release cycles effectively.

Required Technical Skills :

- Cloud & Tools : Azure Data Factory, Databricks, Azure ecosystem

- Databases : SQL Server, T-SQL, PL/SQL

- Reporting & BI Tools : Power BI (PBI)

- Data Concepts : Data Warehousing, ETL/ELT, Data Cleansing, Data Migration

- Other : Version control, Agile methodologies, good problem-solving skills

Preferred Qualifications :

- Solid understanding of cloud data architecture and analytics processes

- Ability to independently initiate and lead conversations with business stakeholders