Data Engineer - Python/ETL

As a Data Exchange Platform Developer, your role involves collaborating with city officials, agencies, system integrators, solution providers, and other data sources to integrate relevant datasets into the data exchange platform. Your responsibilities include deploying and managing the data exchange while supporting application providers in building solutions for optimized management and citizen services. You will evaluate APIs and datasets, create data models, develop software ETL modules, perform unit testing, and deploy them in cloud environments. Additionally, you will develop ETL modules in Python to ingest data into the data exchange using REST APIs and streaming protocols like AMQP and MQTT. This also includes containerizing the adapters, creating data models, and catalog entries according to data exchange specifications. It is essential to follow best practices for software development and adhere to Agile methodology. Key Responsibilities: - Collaborate with various stakeholders to integrate relevant datasets into the data exchange platform - Deploy and manage data exchange while supporting application providers in building solutions - Evaluate APIs and datasets, create data models, and develop software ETL modules - Develop ETL modules in Python for data ingestion using REST APIs and streaming protocols - Containerize adapters and create data models and catalog entries according to specifications Qualifications Required: - Excellent software design and programming skills in Python, Java/JavaScript - Proficiency in data model technologies such as JSON-schema, JSON-LD, and Protobuf - Strong understanding of available data sources and APIs in city solutions - Experience in developing ETL modules for data ingestion and containerization - Proficient in Linux, GIT version control, and cloud computing platforms - Knowledge of geospatial data processing using GIS tools - Understanding of IoT, GIS, Big Data, and Cloud applications - Experience in containerization using Docker and familiarity with Kubernetes - Knowledge of monitoring and logging tools like Prometheus, Grafana, and Logstash would be advantageous This role offers the opportunity to work with cutting-edge technologies and contribute to creating a positive societal impact by improving operational efficiency and service delivery.,