Data Engineer

Job Description

Job Summary: We are looking for a skilled Data Engineer to join our data engineering team. The ideal candidate will have hands-on experience building and optimizing data pipelines, writing complex SQL queries, and working with cloud-based big data technologies such as Azure Databricks , PySpark , and Python . Experience with Azure or AWS cloud platforms is essential. Key Responsibilities: Design, develop, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. Work with Azure Databricks or AWS Glue/EMR to create and maintain ETL workflows using PySpark and Python . Write complex SQL queries to extract and manipulate data from relational and non-relational data sources. Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements. Optimize performance of large-scale data processing systems and ensure data quality, integrity, and governance. Deploy and monitor data pipelines in a cloud-native environment (Azure or AWS). Implement best practices for data modeling, metadata management, and data security. Required Skills & Qualifications: 3+ years of experience as a Data Engineer or in a similar role. Proficiency in Python and SQL for data manipulation and scripting. Strong hands-on experience with Databricks and PySpark . Experience with Azure (e.g., Data Factory, Data Lake, Synapse) or AWS (e.g., S3, Glue, Redshift). Knowledge of data warehousing concepts and distributed systems. Familiarity with CI/CD for data workflows and version control systems like Git. Experience with data pipeline orchestration tools is a plus (e.g., Airflow, Azure Data Factory).