Data Engineer

As a Data Engineer at the company, you will play a crucial role in designing, developing, and maintaining scalable and efficient big data processing pipelines for distributed computing systems. You will collaborate with cross-functional teams to understand data requirements and design appropriate data solutions. Your key responsibilities will include: - Implementing data ingestion, processing, and transformation processes to support various analytical and machine learning use cases. - Optimizing and tuning data pipelines for performance, scalability, and reliability. - Monitoring and troubleshooting pipeline performance issues, identifying and resolving bottlenecks. - Ensuring data quality and integrity throughout the pipeline, and implementing data validation and error handling mechanisms. - Staying updated on emerging technologies and best practices in big data processing and analytics, and incorporating them into our data engineering practices. - Documenting design decisions, technical specifications, and data workflows. Qualifications required for this role include: - Expertise in Data modelling, Data warehousing concepts, data governance best practices, and ETL processes. - Understanding of distributed computing principles and experience with distributed data processing frameworks like Apache Spark or Hadoop. - Familiarity with containerization technologies like Docker and Orchestration tools like Kubernetes. - Ability to develop data models, schemas, and structures to support business needs. - Good understanding of a DW platform like Snowflake, RedShift, etc.,