Principal Data Engineer - ETL Development

Job Description

Job Responsibilities : 1. ETL Development and Data Processing : - Design, develop, and implement ETL processes to extract, transform, and load data from various sources into data warehouses or analytics systems. - Maintain and enhance ETL pipelines to ensure data quality, integrity, and consistency. 2. DevOps and CI/CD Pipeline Management : - Implement and support CI/CD pipelines for Enterprise Data Warehouse (EDW) code development and deployment using GitLab. - Manage and configure workflows for code repositories, ensuring seamless code integration and promotion across environments. 3. Data Analysis and Reporting : - Apply statistical analysis and data mining techniques to uncover trends, patterns, and insights within large datasets. - Develop and maintain dashboards, reports, and visualizations to communicate key insights to stakeholders. 4. Programming and Scripting : - Develop and maintain Linux shell scripting, Python, and PL/SQL code for data extraction, transformation, and automation tasks. - Create efficient scripts for data replication, migration, and data integration between systems. 5. Quality Assurance and System Optimization : - Perform quality assurance checks on system installations, configurations, and code to ensure performance and scalability. - Recommend improvements to optimize performance and reliability of data processes and systems. 6. Business Understanding and Translation : - Work closely with business stakeholders to understand their goals, processes, and requirements, translating them into technical data solutions. - Ensure alignment between technical solutions and business objectives to drive strategic decision-making. 7. Data Governance and Security : - Implement data governance best practices to ensure the security, integrity, and privacy of enterprise data. - Adhere to data security regulations and policies, ensuring that data is handled with confidentiality and compliance. 8. Collaboration and Stakeholder Communication : - Collaborate with cross-functional teams, including data analysts, engineers, and business teams, to deliver impactful data solutions. - Present data findings and actionable insights to stakeholders at various organizational levels, both written and verbally. Required Qualifications : Experience : - 8+ years of experience in IT with a focus on Business Intelligence, Data Warehousing, and Analytics systems. - Proven experience with DevOps methodologies, CI/CD pipelines, and GitLab workflows. - Proficient in ETL development and creating data workflows in SQL, PL/SQL, Python, and Linux shell scripting. Technical Skills : - Strong knowledge of open-source technologies, Linux, and GitLab for codebase management. - Experience in data replication and data migration processes. - Hands-on experience with Python and Node.js for programming and automation tasks. Soft Skills : - Excellent problem-solving and analytical abilities. - Strong communication skills with the ability to convey complex technical concepts to non-technical stakeholders. - Ability to manage multiple projects in a dynamic and fast-paced environment. Preferred Skills : - Experience with data governance principles and ensuring data security compliance. - Familiarity with data quality assurance and optimization techniques. - Understanding of modern data platforms such as cloud-based data warehouses and data lakes. Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.