Job
Description
Responsibilities Create and manage scalable data pipelines to collect, process, and store large volumes of data from various sources Integrate data from multiple sources, ensuring consistency, quality, and reliability Design, implement, and optimize database schemas and structures to support data storage and retrieval Develop and maintain ETL (Extract, Transform, Load) processes to accurately and efficiently move data between systems Build and maintain data warehouses to support business intelligence and analytics needs Optimize data processing and storage performance for efficient resource utilization and quick retrieval Create and maintain comprehensive documentation for data pipelines, ETL processes, and database schemas Monitor data pipelines and systems for performance and reliability, troubleshooting and resolving issues as they arise Stay up to date with emerging technologies and best practices in data engineering, evaluating and recommending new tools as appropriate Requirements Bachelor's or Master's degree in Computer Science, Information Technology, or a related field (Engineering or Math preferred) 5+ years of experience with SQL, Python, .NET, SSIS, and SSAS 2+ years of experience with Azure cloud services, particularly SQL Server, ADF, Azure Databricks, ADLS, Key Vault, Azure Functions, and Logic Apps, with an emphasis on Databricks 2+ years of experience using Git and deploying code using a CI/CD approach Strong analytical and problem-solving skills Excellent communication and interpersonal skills Ability to work independently and as part of a team Attention to detail and a commitment to quality
