Principal Data Engineer (Java | AWS |SPARK |KAFKA )

Role Overview:

Our Engineering team is driving the future of cloud securitydeveloping one of the worlds largest, most resilient cloud-native data platforms. At Skyhigh Security, were enabling enterprises to protect their data with deep intelligence and dynamic enforcement across hybrid and multi-cloud environments. As we continue to grow, were looking for a Principal Data Engineer to help us scale our platform, integrate advanced AI/ML workflows, and lead the evolution of our secure data infrastructure.

Responsibilities:

• As a Principal Data Engineer, you will be responsible for:
• Leading the design and implementation of high-scale, cloud-native data pipelines for real-time and batch workloads.
• Collaborating with product managers, architects, and backend teams to translate business needs into secure and scalable data solutions.
• Integrating big data frameworks (like Spark, Kafka, Flink) with cloud-native services (AWS/GCP/Azure) to support security analytics use cases.
• Driving CI/CD best practices, infrastructure automation, and performance tuning across distributed environments.
• Evaluating and piloting the use of AI/LLM technologies in data pipelines (e.g., anomaly detection, metadata enrichment, automation).
• Evaluate and integrate LLM-based automation and AI-enhanced observability into engineering workflows.
• Ensure data security and privacy compliance.
• Mentoring engineers, ensuring high engineering standards, and promoting technical excellence across teams.

What Were Looking For (Minimum Qualifications)

• 10+ years of experience in big data architecture and engineering, including deep proficiency with the AWS cloud platform.
• Expertise in distributed systems and frameworks such as Apache Spark, Scala, Kafka, Flink, and Elasticsearch, with experience building production-grade data pipelines.
• Strong programming skills in Java for building scalable data applications.
• Hands-on experience with ETL tools and orchestration systems.
• Solid understanding of data modeling across both relational (PostgreSQL, MySQL) and NoSQL (HBase) databases and performance tuning.

What Will Make You Stand Out (Preferred Qualifications)

• Experience integrating AI/ML or LLM frameworks (e.g., LangChain, LlamaIndex) into data workflows.
• Experience implementing CI/CD pipelines with Kubernetes, Docker, and Terraform.
• Knowledge of modern data warehousing (e.g., BigQuery, Snowflake) and data governance principles (GDPR, HIPAA).
• Strong ability to translate business goals into technical architecture and mentor teams through delivery.
• Familiarity with visualization tools (Tableau, Power BI) to communicate data insights, even if not a primary responsibility.

#LI-MS1