Data Engineer

As a Data Engineer at ERM, you will play a crucial role in supporting cutting-edge climate science by engineering high-performance data pipelines and workflows in Azure and Microsoft Fabric. Your responsibilities will include: - Developing, testing, and maintaining scalable data pipelines in Azure and Microsoft Fabric to process climate and environmental data. - Supporting the application of scientific algorithms to global climate model outputs and other multidimensional datasets such as NetCDF and Zarr. - Optimizing data pipelines for performance, scalability, and reliability, including benchmarking storage read/write speeds and tuning Spark-based workflows. - Implementing logging, monitoring, and metadata management systems for traceability and auditability. - Evaluating and configuring Azure storage (Blob, Data Lake, etc.) for climate-scale workloads and ensuring performance alignment with data access needs. - Building and managing distributed processing pipelines to support large-scale data transformations using PySpark and parallel computing strategies. - Monitoring and optimizing I/O performance across workflows and storage tiers. - Collaborating with climate scientists and software engineers to translate prototype code into scalable, production-ready processes. - Documenting data engineering workflows and supporting internal data governance efforts. Qualifications required for this role include: - Bachelors degree in Computer Science, Data Science, Environmental/Climate Science, Oceanography, or a related field. - 2+ years of experience in data engineering, including pipeline development and workflow optimization (cloud-based preferred). - Hands-on experience with Microsoft Azure data services and/or Microsoft Fabric. - Strong programming skills in Python; experience with Spark (PySpark or SparkSQL) in distributed environments. - Demonstrated experience working with large, multidimensional datasets (Zarr, NetCDF, HDF5). - Familiarity with cloud storage performance tuning and benchmarking techniques. - Experience implementing logging, monitoring, and metadata practices in data environments. - Strong communication skills, with the ability to collaborate across technical and non-technical teams. - Preferred: Azure Data Engineering certification, or demonstrable equivalent experience. Join ERM's team of experts in climate science, software development, and environmental consulting, and contribute to making a lasting, positive impact on the planet.,