Cloud Data Engineer

Job Description

End-to-End Data Engineering We are seeking a skilled and hands-on Cloud Data Engineer with 5-8 years of experience to drive end-to-end data engineering solutions. The ideal candidate will have a deep understanding of dimensional modeling, data warehousing (DW), Lakehouse architecture, and the Medallion architecture. This role will focus on leveraging Azure's/AWS ecosystem to build scalable, efficient, and secure data solutions. You will work closely with customers to understand requirements, create technical specifications, and deliver solutions that scale across both on-premise and cloud environments. Key Responsibilities: End-to-End Data Engineering Lead the design and development of data pipelines for large-scale data processing, utilizing Azure/AWS tools such as Azure Data Factory, Azure Synapse, Azure functions, Logic Apps , Azure Databricks, and Data Lake Storage. Tools, AWS Lambda, AWS Glue Develop and implement dimensional modeling techniques and data warehousing solutions for effective data analysis and reporting. Build and maintain Lakehouse and Medallion architecture solutions for streamlined, high-performance data processing. Implement and manage Data Lakes on Azure/AWS, ensuring that data storage and processing is both scalable and secure. Handle large-scale databases (both on-prem and cloud) ensuring high availability, security, and performance. Design and enforce data governance policies for data security, privacy, and compliance within the Azure ecosystem. Customer Interaction & Technical Documentation Interact with clients and business stakeholders to gather and analyze data requirements for building customized solutions. Create clear and concise technical specification documents, detailing the architecture, data flow, and integration plans for project delivery. CI/CD & Automation Implement and manage CI/CD pipelines for data engineering projects, ensuring continuous integration and delivery of data processing and ETL jobs. Automate data workflows and operationalize data processes, ensuring high performance and reliability. Leadership & Mentorship Lead and mentor junior data engineers, fostering a collaborative environment for learning and development. Provide technical leadership and guidance throughout the project lifecycle, ensuring best practices are adhered to in all stages. Required Skills & Experience: Experience: 5-8 years in data engineering roles, with preferably at least 2 years in a lead role. Azure Ecosystem: In-depth experience with Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, and Data Lake Storage. Data Engineering Concepts: Strong understanding of end-to-end data engineering concepts, including ETL pipelines, data integration, and real-time data processing. Dimensional Modeling & Data Warehousing: Solid experience with dimensional modeling and designing scalable data warehousing solutions. Lakehouse Architecture & Medallion Architecture: Practical experience with implementing Lakehouse architecture and Medallion architecture patterns on Azure. Security & Governance: Experience designing data governance frameworks, ensuring data security and compliance with industry standards. CI/CD: Proficiency in setting up and maintaining CI/CD pipelines, automating deployment processes for data engineering. On-prem & Cloud Databases: Experience with managing both on-premise and cloud-based large-scale databases, ensuring performance, security, and scalability. Customer Interaction: Excellent communication skills with the ability to gather business requirements, create technical specs, and ensure stakeholder satisfaction. Preferred Skills: Programming: Experience with programming languages such as Python,SparkSQL for data processing and automation. Big Data Tools: Knowledge of Hadoop, Spark, or other big data processing frameworks. Certifications: Azure/AWS Data Engineer or similar certifications are a plus. Personal Attributes: Problem-Solving: Strong analytical and troubleshooting skills. Collaborative: Ability to work effectively with cross-functional teams and mentor junior engineers. Detail-Oriented: Strong attention to detail with a practical approach to complex data engineering challenges. Salary Range : between 20 - 26 lacs (INR) Time Preferred : Night shift till 1:30 am - IST PTO : 18 days/year and 10 public holidays Important to have very good conversational skills in English Show more Show less