Data Engineer

Core skills and Competencies

1. Design, develop, and maintain data pipelines, ETL/ELT processes, and data integrations to support efficient and reliable data ingestion, transformation, and loading.

2. Collaborate with API developers, and other stakeholders to understand data requirements and ensure the availability, reliability, and accuracy of the data.

3. Optimize and tune performance of data processes and workflows to ensure efficient data processing and analysis at scale.

4. Implement data governance practices, including data quality monitoring, data lineage tracking, and metadata management.

5. Work closely with infrastructure and DevOps teams to ensure the scalability, security, and availability of the data platform and data storage systems.

6. Continuously evaluate and recommend new technologies, tools, and frameworks to improve the efficiency and effectiveness of data engineering processes.

7. Collaborate with software engineers to integrate data engineering solutions with other systems and applications.

8. Document and maintain data engineering processes, including data pipeline configurations, job schedules, and monitoring and alerting mechanisms.

9. Stay up-to-date with industry trends and advancements in data engineering, cloud technologies, and data processing frameworks.

10. Provide mentorship and guidance to junior data engineers, promoting best practices in data engineering and ensuring the growth and development of the team.

11. Able to implement and troubleshoot Rest services in Python.

External Skills And Expertise

1. Strong proficiency in Python or another programming language commonly used in data engineering, such as Java.

2. Excellent SQL skills with ability to work with data across different SQL databases including Postgres, Databricks, SQL Server etc. NoSQL databases is a plus

3. In-depth knowledge of cloud platforms such as AWS, Azure, or Google Cloud, and experience with the services and tools they offer for data storage, processing, and analytics (e.g., S3, Redshift, BigQuery, Dataflow).

4. Experience in building and optimizing ETL/ELT processes, data pipelines, and workflows, using tools like Apache Nifi, or Apache Kafka.

5. Familiarity with data modeling, data warehousing concepts, and data governance best practices.

6. Strong problem-solving and troubleshooting skills, with the ability to identify and resolve data quality and performance issues.

7. Experience with version control systems, such as Git, and knowledge of CI/CD practices in a data engineering context.

8. Knowledge of data security and privacy principles, as well as experience implementing data access controls and managing sensitive data.

10. Knowledge of elastic Search/log stash implementation skills very desirable.

11. Understanding of distributed computing principles and experience with distributed data processing frameworks like Apache Spark or Hadoop.

12. Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.

13.Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and translate business requirements into technical solutions.