Senior Engineer (Data Platform)

Job Description

Optimize Spark clusters for cost, efficiency and performance by implementing robust monitoring systems to identify bottlenecks using data and metrics. Provide actionable recommendations for continuous improvement. Optimize the infrastructure required for extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies . Work with data and analytics experts to strive for greater cost efficiencies in our data systems. Experience with processing large workloads and complex code on Spark clusters. Proven experience in setting up monitoring for Spark clusters and driving optimization based on insights and findings. Experience in designing and implementing scalable Data Warehouse solutions to support analytical and reporting needs. Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency, and workload management. Working knowledge of message queuing, stream processing, and highly scalable big data data stores. Desired Skills & Experience: 4+ years of experience in Data Platform. Expertise with Python and Jupyter notebooks is a must. Experience with big data tools: Spark, Kafka, etc. Experience with relational SQL and NoSQL databases, including Postgres and Cassandra. Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc. Experience with AWS cloud services: EC2, EMR, RDS, Redshift. Working knowledge of stream-processing systems: Storm, Spark-Streaming, etc. is a plus Our Culture: We have an autonomous and empowered work culture encouraging individuals to take ownership and grow quickly. Flat hierarchy with fast decision making and a startup-oriented get things done culture. A strong, fun & positive environment with regular celebrations of our success. We pride ourselves in creating an inclusive, diverse & authentic environment