Pyspark Developer

Hi All,

We are hiring of Pyspark Developer for MNCs Company.

Exp- 5+ Years

Location- Bangalore/Hyderabad/Pune

Immmediate Joiner

Job Description:

We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics.

Responsibilities:

• Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation.
• Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources.
• Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.
• Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.

• Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.

• Ensure data quality and integrity throughout all stages of data processing.

• Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.

• Troubleshoot and debug issues related to data pipelines and processing.

• Ensure compliance with regulatory requirements and industry standards in all data processing activities.

• Implement best practices for data security, compliance, and privacy within Azure environment.Document technical specifications, data flows, and solution architecture.

Requirements:

• Bachelors degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals.
• Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF).
• Proficiency in Python programming language and solid understanding of SQL.
• Experience designing and optimizing data pipelines for ETL/ELT processes.
• Familiarity with cloud platforms, preferably Microsoft Azure.
• Excellent problem-solving skills and ability to think critically.
• Strong communication skills with the ability to collaborate effectively in a team environment.
• Experience in Financial, Risk, Compliance, or Banking domains is a plus.

• Experience identifying and mitigating financial risks in data processes.

• Ability to analyse data for potential risk factors and develop strategies to minimize financial risk.

• Ensure all data processes comply with relevant regulatory requirements and industry standards.

Preferred Qualifications:

• Certification in Azure Data Engineering or related field.
• Knowledge of other big data technologies such as Hadoop, Hive, or Kafka.

• Familiarity with machine learning frameworks and techniques.

priya.singhal@thehrsolutions.in

Regards,

Priya Singhal