Data Engineer II

• Data Pipeline Operations - Design, build, and maintain robust and scalable data pipelines to ingest, transform, and deliver structured and unstructured data from multiple sources. Ensure high-quality data by implementing monitoring, validation, and error-handling processes.
• Platform Engineering & Optimization - Create and update data models to represent the structure of the data. Design, implement, and maintain database systems. Optimize database performance and ensure data integrity. Troubleshoot and resolve database issues. Build and manage data warehouses for storage and analysis of large datasets. Collaborate on data modeling, schema design, and performance optimization for large-scale datasets.
• Data Quality and Governance: Implement and enforce data quality standards. Contribute to data governance processes and policies.
• Scripting and Programming: Develop and automate data processes through programming languages (e.g., Python, Java, SQL). Implement data validation scripts and error handling mechanisms.
• Version Control: Use version control systems (e.g., Git) to manage codebase changes for data pipelines.
• Monitoring and Optimization: Implement monitoring solutions to track the performance and health of data systems. Optimize data processes for efficiency and scalability.
• Cloud Platforms: Work with cloud platforms (e.g., AWS, Azure, GCP) to deploy and manage data infrastructure. Utilize cloud-based services for data storage, processing, and analytics.
• Security: Implement and adhere to data security best practices. Ensure compliance with data protection regulations..
• Troubleshooting and Support: Provide support for data-related issues and participate in root cause analysis.

Skills:

• Expertise in data modeling, database design, and data warehousing. Proficient in SQL and programming languages such as Python, Java, or Scala.
• Experience with big data technologies such as Hadoop, Spark, and Kafka.
• Cloud-native architecture expertise (AWS, GCP, or Azure), including containerization (Docker, Kubernetes) and infrastructure-as-code (Terraform, CloudFormation).