Data Engineer

As a Data Engineer with 4-5 years of experience, you will be responsible for designing and developing scalable data infrastructure and systems to support critical business functions. Your role will involve building robust ETL pipelines, managing large-scale datasets, and optimizing data workflows in a fast-paced environment. Key Responsibilities: - Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data. - Model and structure data for performance, scalability, and usability. - Work with cloud infrastructure, preferably Azure, to optimize data workflows. - Build and manage data lake/lakehouse architectures following best practices. - Optimize ETL performance and ensure cost-effective data operations. - Collaborate with cross-functional teams including data science, analytics, and software engineering. - Ensure data quality, integrity, and security throughout the data lifecycle. Required Skills & Qualifications: - 4-5 years of relevant experience in data engineering. - Advanced proficiency in Python, including Pandas and NumPy. - Strong skills in SQL for complex data manipulation and analysis. - Hands-on experience with distributed systems like Apache Spark or Hadoop. - Track record of handling large-scale datasets in production environments. - Cloud development experience with Azure (preferred), AWS, or GCP. - Understanding of data lake and data lakehouse architectures. - Expertise in ETL performance tuning and cost optimization. - Knowledge of data structures, algorithms, and software engineering practices. Soft Skills: - Strong communication skills to explain technical concepts clearly. - Self-starter mindset with a quick learning ability and ownership. - Attention to detail with a focus on data quality and reliability. - Comfortable working in an agile environment with evolving requirements. Preferred Qualifications: - Experience with tools like Azure Data Factory or similar. - Familiarity with CI/CD and DevOps in the context of data engineering. - Knowledge of data governance, cataloging, and access control principles.,