Urgently Need -- Principal Data Engineer

Profile:

As a Principal Data Engineer, to lead the architecture, design, and implementation of scalable and efficient data pipelines, data models, and data infrastructure solutions on GCP. To collaborate with cross-functional teams, including Data Scientists, Analysts, and Product Engineers, to build data architectures that support data-driven decision-making across the organization.

Also, to play a crucial role in the data engineering team's efforts to manage and optimize data infrastructure, ensuring performance, reliability, and scalability on GCP

Key Responsibilities

Architect Data Solutions:

• Lead the design and implementation of complex data architectures and pipelines on Google Cloud Platform (GCP) to meet business requirements.

Cloud Infrastructure Management:

• Build and maintain cloud-based data solutions using GCP services such as BigQuery, Dataflow, Dataproc, Pub/Sub, GCS, Composer, and others.

Data Modelling and Optimization:

• Develop and maintain efficient, scalable data models, ensuring optimal data storage, retrieval, and processing.

Pipeline Development:

• Lead the creation and optimization of ETL/ELT pipelines that integrate diverse data sources into centralized data repositories.

Collaboration:

• Work closely with stakeholders from data science, business intelligence, analytics, and product teams to deliver end-to-end data solutions.

Leadership:

• Mentor and guide junior data engineers and provide technical leadership in architecture, development, and best practices in cloud data engineering.

Automation and Monitoring:

• Ensure automated data workflows, implement monitoring tools, and resolve issues in data pipelines to ensure data availability, integrity, and accuracy.

Security and Compliance:

• Implement and ensure best practices for data security, privacy, and compliance with relevant data protection regulations on GCP.

Qualifications

• Bachelors Degree or above in Computer Science, Information Technology, Engineering, Mathematics, or a related field.
• Relevant certifications in Cloud technologies, particularly Google Cloud Professional Data Engineer or other GCP-specific certifications, are a plus.
• 7+ years of experience as a Data Engineer or in a similar role, with a strong background in cloud data platforms, particularly GCP.

Skills and Competencies

GCP Expertise:

• Hands-on experience with Google Cloud Platform services, including: BigQuery, Dataflow, Pub/Sub, Dataproc, Composer, GCS, and Cloud Functions
• Experience with GCP's security features, cost optimization, and best practices for data management.

Data Engineering Skills:

• Expertise in designing, developing, and deploying large-scale data pipelines and data architectures in a cloud environment.

Programming Languages:

• Proficiency in Python, SQL, and/or Java for developing data pipelines, transformations, and automation tasks.

Data Modeling:

• Strong experience with relational and NoSQL databases, schema design, and dimensional modeling.

ETL Tools:

• Experience with ETL tools like Apache Beam, Airflow, or similar for building data pipelines.

Data Warehousing:

• Experience with data warehousing concepts and tools, particularly BigQuery or similar cloud data warehousing technologies.

Version Control:

• Proficiency in using version control systems such as Git.

CI/CD:

• Familiarity with Continuous Integration and Continuous Deployment (CI/CD) practices for data engineering pipelines.

Leadership:

• Demonstrated experience in leading a team, mentoring engineers, and contributing to a collaborative work environment.

Problem-Solving:

• Strong analytical and troubleshooting skills, with the ability to solve complex data problems.