Data Engineer - Contract - Remote

Job Description

Project Name: Infrastructure Discovery Client: LSEG (London Stock Exchange Group) Objective: Build a solution to manage the lifecycle of infrastructure devices, particularly those approaching end-of-life. Develop a recommendation engine for devices nearing end-of-life. Ensure operational efficiency and support for the infrastructure discovery solution. Key Responsibilities: Data Integration and Management: Pull data from various sources through APIs and manual entries. Maintain and upload manual files as needed. Ensure data pipelines are operational and do not break. Development and Optimization: Develop and optimize Python code to implement logic and rules for the recommendation engine. Ensure the code runs smoothly and make small tweaks as needed. Implement data transformation and loading processes. Dashboard and Reporting: Connect the Python code to Power BI to display results and recommendations on a dashboard. Ensure the pipeline from the analytical engine to the dashboards is operational. Support and Maintenance: Provide ongoing support to keep the solution operational. Perform light feature modifications based on user feedback. Ensure the analytical engine and dashboards are working correctly. Collaboration and Communication: Work closely with the management consulting team and other stakeholders. Translate client requirements into technical specifications. Communicate effectively with the support team to manage user feedback and requests. Technical Stack: Programming Languages: Python Data Visualization: Power BI Cloud Services: AWS (Lambda, Glue, S3, Step Function, CloudWatch, ECS) Data Management: SQL, Snowflake Workflow Orchestration: Apache Airflow Desired Skills and Experience: Technical Skills: Proficiency in Python programming. Experience with data integration and management. Familiarity with Power BI for data visualization. Knowledge of cloud services (AWS) and workflow orchestration (Apache Airflow). Problem-Solving: Ability to troubleshoot and optimize data pipelines. Experience in making small tweaks and modifications to existing code. Collaboration: Strong communication skills to interact with cross-functional teams and stakeholders. Ability to translate client requirements into technical specifications. Support and Maintenance: Experience in providing ongoing support and maintenance for data solutions. Ability to handle user feedback and make necessary adjustments.