Project Partner

Job Summary:

We are looking for a skilled and detail-oriented Associate Data Engineer with 3 to 8 years of professional experience to join our growing data team. The candidate should be proficient in designing data models, managing databases, developing scalable data pipelines (especially using Azure Data Factory), and programming in Python, PySpark, and SQL. The role involves end-to-end development, optimization, and maintenance of data workflows that support critical business functions.

Key Responsibilities:

• Data Modelling & Documentation

o Design, implement, and maintain efficient data models for structured and unstructured data.

o Create and maintain detailed technical documentation for data flows, architecture, and transformations.

• Database Handling & Development

o Manage large-scale databases, including schema design, indexing, and performance tuning.

o Write complex stored procedures, triggers, and functions to support data workflows.

• Data Pipeline Development

o Build and manage robust, scalable ETL/ELT pipelines using Azure Data Factory (ADF).

o Integrate diverse data sources including APIs, flat files, databases, and cloud services.

• Programming & Scripting

o Develop and maintain data transformation scripts using Python, PySpark, and SQL.

o Optimize code for performance and scalability in distributed environments.

• Testing & Validation

o Conduct thorough unit, integration, and regression testing for data solutions.

o Implement data quality checks, monitoring, and alerting mechanisms.

• Cross-functional Collaboration

o Collaborate with data analysts, data scientists, and business teams to gather requirements and deliver solutions.

o Translate complex business needs into efficient and scalable technical implementations.

Required Skills & Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 3 to 8 years of experience in data engineering, ETL development, or related fields.
• Strong proficiency in SQL, Python, and PySpark.
• Hands-on experience with Azure Data Factory and cloud-based data integration services.
• Solid understanding of data modelling, database design, and data warehousing concepts.
• Experience working with RDBMS (e.g., SQL Server, PostgreSQL, Oracle).
• Knowledge of version control tools like Git.
• Strong analytical and problem-solving skills.
• Excellent communication and documentation skills.

Preferred Qualifications:

• Experience with cloud platforms (Azure, AWS, or GCP).
• Understanding of CI/CD and DevOps practices for data engineering.