Cribl Data Engineer

Job Description

Job Summary: We are looking for a skilled Data Engineer with experience in ETL, AWS Glue, Snowflake, Cribl, Azure Data Factory(ADF), Azure Databricks(ADB), Azure fundamentals, Python, and PySpark. The ideal candidate should have strong experience in designing, building, and maintaining scalable data pipelines while ensuring data integrity, security, and performance optimization. Cribl Stream, Edge and Search and Snowflake knowledge is must to have, AWS Glue is value add. Data Pipeline Development Design, develop, and maintain ETL/ELT pipelines using ADF, ADB or AWS Glue and PySpark. Extract, transform, and load (ETL) data from various sources into Snowflake. Optimize data processing workflows for performance and cost-efficiency. Data Integration & Processing: Work with Cribl to manage data ingestion and transformation. Integrate structured, semi-structured (JSON, Avro, Parquet), and unstructured data sources. Database & Cloud Management: Implement data modeling, partitioning, and indexing strategies in Snowflake. Ensure secure and efficient data storage, transformation, and access. Leverage AWS services (S3, Lambda, IAM, Step Functions) for data operations. Performance Optimization & Monitoring: Optimize PySpark jobs for scalability and cost efficiency. Monitor job execution, error handling, and performance tuning. Implement data quality and validation checks across pipelines. Collaboration & Documentation: Work closely with data analysts, BI teams, and DevOps engineers to support analytics needs. Document ETL processes, data flows, and best practices for future reference.