DATA ENGINEER

Requirements

• Experience with processing large workloads and complex code on Spark clusters.
• Experience setting up monitoring for Spark clusters and driving optimization based on insights and findings.
• Understanding designing and implementing scalable data warehouse solutions to support analytical and reporting needs.
• Strong analytic skills related to working with unstructured datasets.
• Understanding of building processes supporting data transformation, data structures, metadata, dependency, and workload management.
• Knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Knowledge of Python and Jupyter Notebooks.
• Knowledge of big data tools like Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
• Experience with data pipeline and workflow management tools like Azkaban, Luigi, Airflow, etc.
• Experience with AWS cloud services (EC2, EMR, RDS, and Redshift).
• Willingness to work from an office at least 2 times per week.
  

Responsibilities

• Optimize Spark clusters for cost, efficiency, and performance by implementing robust monitoring systems to identify bottlenecks using data and metrics. Provide actionable recommendations for continuous improvement.
• Optimize the infrastructure required for extracting, transforming, and loading data from various data sources using SQL and AWS ‘big data’ technologies.
• Work with data and analytics experts to strive for greater cost efficiencies in the data systems.
  

• US and EU projects based on advanced technologies.
• Competitive compensation based on skills and experience.
• Annual performance appraisals.
• Remote-friendly culture and no micromanagement.
• Personalized learning program tailored to your interests and skill development.
• Bonuses for article writing, public talks, other activities.
• 15 PTO days, 10 national holidays.
• Free webinars, meetups and conferences organized by Svitla.
• Fun corporate celebrations and activities.
• Awesome team, friendly and supportive community!