Job
Description
As a Data Engineer you would be working to design, build, test and create infrastructure necessary for real time analytics and batch analytics pipelines. You will work with multiple teams within the org to provide analysis, insights on the data. You will also be involved in writing ETL processes that support data ingestion. You will also guide and enforce best practices for data management, governance and security. You will build infrastructure to monitor these data pipelines / ETL jobs / tasks and create tooling/infrastructure for providing visibility into these. Responsibilities We are looking for a Data Engineer with passion for building data pipelines, working with product, data science and business intelligence teams and delivering great solutions. As a part of the team you:- Acquire deep business understanding on how SPAN data flows from IoT device to cloud through the system and build scalable and optimized data solutions that impact many stakeholders. Be an advocate for data quality and excellence of our platform. Build tools that help streamline the management and operation of our data ecosystem. Ensure best practices and standards in our data ecosystem are shared across teams. Work with teams within the company to build close relationships with our partners to understand the value our platform can bring and how we can make it better. Improve data discovery by creating data exploration processes and promoting adoption of data sources across the company. Have a desire to write tools and applications to automate work rather than do everything by hand. Assist internal teams in building out data logging, alerting and monitoring for their applications Are passionate about CI/CD process. Design, develop and establish KPIs to monitor analysis and provide strategic insights to drive growth and performance. Required Qualifications Bachelors Degree in a quantitative discipline: computer science, statistics, operations research, informatics, engineering, applied mathematics, economics, etc 7+ years of relevant work experience in data engineering, business intelligence, research or related fields. Expert level production-grade, programming experience in at least one of these languages (Python, Kotlin, or other JVM based languages) Experience in writing clean, concise and we'll structured code in one of the above languages. Experience working with Infrastructure-as-code tools: Pulumi, Terraform, etc Experience working with CI/CD systems: Circle-CI, Github Actions, Argo-CD, etc Experience managing data engineering infrastructure through Docker and Kubernetes Experience working with latency data processing solutions like Flink, Prefect, AWS Kinesis, Kafka, Spark Stream processing etc Experience with SQL/Relational databases, OLAP databases like Snowflake. Experience working in AWS: S3, Glue, Athena, MSK, EMR, ECR etc Bonus Qualifications Experience with the Energy industry Experience with building IoT and/or hardware products Understanding of electrical systems and residential loads Experience with data visualization using Tableau. Experience in Data loading tools like FiveTran as we'll as data debugging tools such as DataDog
