Principal Data Engineer

About KYFEX:

KYFEX is a leading AI consulting firm, dedicated to harnessing the power of artificial intelligence to revolutionize business operations across the globe. Our expertise in Large Language Models (LLMs) positions us at the cutting edge of AI technology, enabling us to offer unparalleled solutions to our clients. As we continue to grow, we're seeking a skilled Remote Principal Data Engineer to lead our efforts in managing and optimizing data processes for LLM training.

Job Responsibilities:

• Lead the design and implementation of scalable data pipelines for the training of open-source LLMs.
• Work closely with AI researchers and engineers to understand data requirements and ensure efficient data processing for AI model training.
• Develop and maintain robust data storage solutions, ensuring data integrity, security, and compliance.
• Optimize data retrieval and processing techniques to reduce training time and improve model performance.
• Implement monitoring, logging, and alert systems to ensure high availability and performance of data systems.
• Collaborate with cross-functional teams to integrate LLM solutions into client projects, providing expert advice on data engineering best practices.
• Stay abreast of the latest developments in data engineering and LLM technologies, continuously improving KYFEX’s data strategies.

Minimum Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with a proven track record of building and managing large-scale data pipelines.
• Strong proficiency in programming languages such as Python or Scala.
• Extensive experience with big data technologies (e.g., Hadoop, Spark, Kafka) and cloud services (AWS, Google Cloud, Azure).
• Demonstrated experience in data modeling, ETL development, and data warehousing.
• Knowledge of machine learning concepts and experience supporting data needs for AI/ML projects.
• Excellent problem-solving skills and the ability to work independently in a fully remote environment.

Preferred Skills:

• Master’s degree or Ph.D. in a related field.
• Experience with open-source LLMs and understanding of NLP data processing.
• Familiarity with containerization and orchestration technologies (Docker, Kubernetes).
• Experience in implementing data security and privacy practices.
• Strong communication skills, with the ability to lead teams and collaborate effectively with stakeholders.

Why Join KYFEX?

• Work at the forefront of AI technology with a team of experts passionate about innovation.
• Enjoy the flexibility and benefits of a fully remote position.
• Engage in challenging and meaningful projects that have a real-world impact.
• Benefit from a culture of continuous learning, professional development, and collaborative achievement.

To Apply:

KYFEX is committed to diversity and inclusion and encourages applications from all qualified individuals, including those from diverse backgrounds and underrepresented groups.