Freelance GCP Data Engineer - Apache NiFi & SQL Expert

Company Description

ThreatXIntel is a startup specializing in cybersecurity, offering tailored and cost-effective solutions for businesses and organizations of all sizes. Our expertise spans cloud security, web and mobile security testing, cloud security assessments, DevSecOps, and more. Driven by a proactive approach, we monitor and test our clients' digital environments to identify vulnerabilities before they can be exploited. Our mission is to deliver reliable and accessible cybersecurity services, empowering businesses to focus on growth while safeguarding their digital assets. Based on a commitment to quality, our team strives to provide peace of mind through customized protection strategies.

Role Description

Freelance GCP Data Engineer

Key Responsibilities

• Design and implement real-time data ingestion pipelines

  using

  Apache NiFi

  for structured and unstructured data.

• Integrate data pipelines

  with the

  GCP ecosystem

  , including

  BigQuery

  ,

  Pub/Sub

  ,

  Dataflow

  ,

  Dataproc

  ,

  Cloud Storage

  , and

  Composer/Airflow

  .
• Ensure

  low-latency, high-throughput data delivery

  to downstream systems for

  analytics

  and

  reporting

  .

• Develop and optimize SQL queries

  for

  BigQuery

  and relational databases to ensure data transformation and analysis.

• Write Python scripts

  for data transformation, automation, and custom

  NiFi processors

  when required.
• Implement

  data quality

  , validation, and

  error-handling mechanisms

  within ingestion pipelines to ensure reliability and accuracy.
• Collaborate with

  data analysts

  ,

  data scientists

  , and

  platform engineers

  to deliver consistent and high-quality datasets.
• Ensure

  data security

  ,

  governance

  , and compliance with

  GCP IAM

  ,

  VPC Service Controls

  , and

  encryption standards

  .

• Monitor, troubleshoot

  , and optimize data pipeline performance to ensure smooth operations.
• Document

  data flows

  , architecture, and best practices to maintain consistency across teams.

Required Skills & Qualifications

• Proven experience

  with

  Apache NiFi

  for real-time ingestion, routing, and transformation of data.
• Strong expertise in the

  GCP ecosystem

  , including:

• BigQuery

  (data warehousing)

• Pub/Sub

  (event streaming)

• Dataflow/Apache Beam

  (data processing)

• Cloud Storage

  (data lake)

• Composer/Airflow

  (workflow orchestration)
• Strong

  SQL skills

  for querying, modeling, and performance optimization of data.
• Proficiency in

  Python

  for data transformation, automation, and creating custom NiFi processors.
• Experience with

  real-time streaming concepts

  , such as windowing, late-arriving data, and deduplication.
• Strong understanding of

  ETL/ELT processes

  ,

  data lakes

  , and

  data warehouses

  .
• Familiarity with

  CI/CD practices

  and

  version control

  using

  Git

  .
• Understanding of

  data security

  ,

  governance

  , and

  compliance

  within cloud environments.