Senior Big Data Engineer

We are seeking an Senior Big Data Engineers to join our team immediately who can design, develop, maintain, and implement large-scale data processing systems for the Sherlock platform. In this role, you will work closely with cross-functional teams such as data scientists, software engineers, business analysts, and product managers to ensure the successful implementation and scaling of our data infrastructure.

As an ideal candidate, you must have a deep understanding of big data technologies and best practices and the ability to work with large amounts of data from various sources. One of the key skills required for this role is the ability to understand existing solutions and debug performance issues. This means being able to identify bottlenecks in the system, analyze the root cause of performance issues, and implement solutions to improve system efficiency.

Responsibilities

Design, develop, and implement large-scale data processing systems using Big Data technologies such as Spark, Cassandra, and Kafka

Develop data pipelines for data ingestion, transformation, and storage

Develop and maintain data models and database schemas for structured and unstructured data

Identify bottlenecks in the system, analyze the root cause of performance issues, and implement solutions to improve system efficiency

Implement data security and access control measures to ensure the confidentiality, integrity, and availability of data

Develop and maintain data quality and validation frameworks to ensure the accuracy and completeness of data

Develop and maintain monitoring and alerting systems to proactively identify and address data processing issues

Work closely with data scientists, software engineers, business analysts, and product managers to understand data requirements and design solutions that meet their needs

Continuously evaluate and implement new Big Data technologies and frameworks to improve the performance, scalability, and reliability of our data infrastructure

Ensure compliance with data privacy regulations such as GDPR and CCPA

Requirements

Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related field

5+ years of experience in Big Data engineering or a related field

Strong programming skills in one or more languages such as Python, Java or Scala

Strong experience with Big Data technologies such as Spark, Cassandra, and Kafka

Strong experience with data modeling and database design for structured and unstructured data

Strong experience with data processing and ETL frameworks such as Apache NiFi, Airbyte, or Apache Beam

Strong experience with data security and access control measures

Strong experience with data quality and validation frameworks

Experience with monitoring and alerting systems for Big Data technologies

Experience with cloud-based Big Data platforms such as Databricks, AWS EMR or Google Cloud Dataproc

Excellent communication and collaboration skills with the ability to work effectively in a cross-functional team environme