Urgent Opening For Senior Advisor - Big Data, Analytics and ML

Key Responsibilities:

Solution Design & Architecture:

• Lead the design and architecture of complex software and data engineering solutions, ensuring they are scalable, reliable, and secure. 
• Build and maintain

  end-to-end data pipelines

   for efficient data ingestion, processing, and transformation in cloud environments (primarily

  Google Cloud

  ). 
• Design and implement cloud-based data solutions using

  Cloud Run

  ,

  Cloud Functions

  ,

  Big Query, Datastore

  ,

  GCS

  ,

  Dataplex

  ,

  Pub/Sub, Cloud Scheduler

  and

  Composer

  . 
• Ensure data quality, consistency, and integrity across data sources and throughout the ETL pipeline. 
• Implement

  data governance

   best practices, ensuring compliance, security, and data privacy in the processing of sensitive information.  

Software Engineering & Development:

• Contribute to the development of scalable, maintainable, and efficient software applications in

  Python, Scala, Spark

  . 
• Write code to integrate various data systems and ensure seamless data flow between storage, processing, and analysis layers. 
• Develop and optimize

  SQL queries, Stored Procedure

   for

  Big Query

   to support efficient data retrieval and analytics. 

Mentorship & Team Guidance:

• Mentor junior software and data engineers, providing guidance on technical aspects (e.g., data pipeline design, cloud technologies, OOP principles). 
• Provide technical leadership in both

  software engineering

   and

  data engineering

  , helping to define best practices and standards for the team. 

Collaboration & Communication:

• Work closely with cross-functional teams to define requirements and deliver high-quality solutions. 
• Communicate technical solutions clearly and concisely 

Code Quality & Best Practices:

• Promote a culture of clean, maintainable code and ensure best practices for software and data engineering are followed. 
• Implement design patterns and architectural principles in both software and data engineering projects to ensure long-term maintainability. 

Continuous Improvement:

• Stay up to date with emerging trends in both

  software/data engineering to

   bring new tools, technologies, and approaches to the team. 
• Advocate for continuous improvement in the development and deployment processes, particularly around data management, cloud services, and automation.  

Required Skills & Qualifications:

• Object-Oriented Programming (OOP)

   and software design principles. 

• Python

   for building cloud-native applications, data processing, and automation. 
• Extensive experience with

  Google Cloud Platform (GCP)

  , particularly with

  BigQuery

  ,

  Cloud Functions

  ,

  Cloud Run

  ,

  Datastore

  ,

  GCS, Pub/Sub, Cloud Scheduler

  ,

  Dataplex

  , and

  Composer

  . 
• Strong expertise in

  data engineering

  , including building

  ETL pipelines

  ,

  data warehousing

  , and

  data storage solutions

  . 
• Proficiency in

  PostgreSQL

  ,

  MS-SQL

  , and designing relational database solutions. 
• Familiarity with

  Apache Airflow

   for workflow orchestration and managing data pipelines. 
• Ability to design scalable

  data architectures

   and work with both structured and unstructured data. 

Architectural Skills:

• Deep understanding of

  design patterns

   and their application to both software engineering and data engineering solutions. 
• Experience designing and implementing large-scale

  data systems

   and distributed architectures. 

Mentorship & Leadership:

• Proven experience mentoring and guiding junior engineers  

Nice to Have:

• Experience with

  Apache Spark and Scala

   for large-scale data processing. 
• Exposure with

  Azure

   cloud services.

Education:

• Bachelor's in computers science or related.