Freelance – Apache NiFi & GCP Data Engineer

Company Description

ThreatXIntel is a dynamic cybersecurity startup dedicated to protecting businesses and organizations against cyber threats. Specializing in services such as cloud security, DevSecOps, and vulnerability testing, the company offers tailored, cost-effective solutions to meet clients' unique requirements. Focused on startups and small businesses, ThreatXIntel ensures that high-quality cybersecurity services are accessible to safeguard digital assets. With a proactive approach, the team continuously monitors and tests digital environments to identify vulnerabilities before threats arise. ThreatXIntel is committed to delivering peace of mind, allowing clients to prioritize business growth.

Role Description

Apache NiFi

Key Responsibilities

• Design and implement

  real-time data ingestion pipelines

  using Apache NiFi for both structured and unstructured datasets.
• Integrate ingestion and processing pipelines with the

  GCP ecosystem

  , including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow).
• Ensure

  low-latency and high-throughput

  data movement across ingestion, processing, and storage layers.
• Develop and optimize

  SQL queries

  for BigQuery and relational databases to support analytics and reporting.
• Build

  Python scripts

  for data transformation, automation, and developing custom NiFi processors.
• Implement

  data quality checks

  , validation rules, deduplication, late-arriving data handling, and robust error-handling frameworks.
• Collaborate with data analysts, data scientists, and platform engineering teams to deliver analytics-ready datasets.
• Apply data security, GCP IAM policies, VPC Service Controls, and encryption standards to maintain compliance.
• Monitor, troubleshoot, and optimize pipeline performance to ensure stability and reliability.
• Document data flows, architecture diagrams, transformation logic, and operational best practices.

Required Skills & Qualifications

• Strong hands-on experience with

  Apache NiFi

  for real-time ingestion, routing, transformation, and orchestration.
• Deep knowledge of key

  GCP services

  , including:

• Pub/Sub

  (event ingestion & streaming)

• BigQuery

  (warehouse, SQL optimization)

• Dataflow / Apache Beam

  (data processing pipelines)

• Dataproc

  (Spark/Hadoop processing)

• Cloud Storage

  (data lake)

• Composer / Airflow

  (workflow orchestration)
• Advanced

  SQL

  skills for querying, optimization, and data modeling.
• Proficiency in

  Python

  for automation, ETL tasks, and scripting custom processors.
• Strong understanding of

  real-time streaming concepts

  such as windowing, watermarks, deduplication, and event-time processing.
• Knowledge of

  data architecture principles

  , including ETL/ELT design, data lakes, and warehouse best practices.
• Experience with

  CI/CD

  , Git-based version control, and collaborative development workflows.
• Familiarity with

  security and governance practices

  in cloud-based data environments.

Ideal Candidate Profile

• Hands-on engineer comfortable owning end-to-end ingestion pipeline development.
• Strong communicator capable of collaborating with diverse technical teams.
• Self-driven, detail-oriented, and capable of delivering high-quality work in fast-paced environments.
• Ability to solve complex data engineering challenges using modern cloud-native tools.