Senior Data Engineer

Job Description

Overview: Key Responsibilities: Design and develop data pipelines and workflows to extract, transform, and load (ETL/ELT) data from various sources into target systems. Automate workflows to ensure efficiency, scalability, and error reduction in data integration processes. Optimize and manage data storage solutions, including data lakes and warehouses, for high performance and scalability. Ensure data quality by implementing processes to validate completeness, accuracy, and consistency of data. Ensure compliance with data governance policies and maintain data privacy and security standards. Collaborate with Data Scientists, Analytics Engineers, and stakeholders to understand business requirements and deliver high-quality data solutions. Implement monitoring and observability tools for data pipelines to ensure reliability and real- time issue detection. Stay abreast of the latest developments and advancements, including new and emerging technologies & best practices and new tools & software applications and how they could impact CDM Smith. Assist with the development of documentation, standards, best practices, and workflows for data technology hardware/software in use across the business. Perform other duties as required. Skills and Abilities: Experience with the Software Development Life Cycle (SDLC) and Agile Development methodologies. Strong expertise in Microsoft Azure cloud services, including Azure Data Factory, Azure Databricks, and Azure Synapse Analytics. Proficiency in building and optimizing data systems using modern frameworks like Apache Spark and Databricks. Expertise in data modeling and designing scalable ETL/ELT processes. Experience with real-time streaming solutions, such as Kafka or Azure IoT Hub. Hands-on experience with distributed computing tools, including Apache Spark, and Hadoop. Familiarity with CI/CD pipelines and DevOps practices for data solutions. Knowledge of monitoring tools and techniques for ensuring pipeline observability and reliability. Excellent problem-solving and critical thinking skills to identify and address technical challenges effectively. Excellent interpersonal and presentation skills to build strategic relationships with colleagues, stakeholders, and partners. Strong critical thinking skills to generate innovative solutions and improve business processes. Ability to effectively communicate complex technical concepts to both technical and non- technical audiences. Detail oriented with the ability to assist with executing highly complex or specialized projects. Minimum Qualifications: Bachelors degree. 6 years of related experience. Equivalent additional directly related experience will be considered in lieu of a degree. Preferred Qualifications: