Data Engineer (SQL, Spark/Pyspark, Databricks)

Summary

Job Profile Summary

Data Engineer

Your role in our mission

• Design, develop and deploy data pipelines including ETL-processes for getting, processing and delivering data using Databricks or Apache Spark Framework.
• Monitor, manage, validate and test data extraction, movement, transformation, loading, normalization, cleansing and updating processes. Build complex databases that are useful, accessible, safe and secure.
• Coordinates with users to understand data needs and delivery of data with a focus on data quality, data reuse, consistency, security, and regulatory compliance.
• Collaborate with team-members on data models and schemas in our data warehouse.
• Collaborate with team-members on documenting source-to-target mapping.
• Conceptualize and visualize data frameworks.
• Communicate effectively with various internal/external stakeholders.

What we're looking for

• Bachelor's degree in computer sciences or related field

• 14+ years of experience or equivalent experince in ETL Processing/data architecture or equivalent
• 6+ years of experience working with big data technologies on AWS/Azure/GCP
• 4+ years of experience in the Databricks/Apache Spark framework (Python/Scala)
• Experience working with different between database structures (e.g., transaction based vs. data warehouse)
• Databricks and AWS developer/architect certifications a big plus
• Ability to lead and mentor a team of engineers         

Other Qualifications

• Strong project planning and estimating skills related to area of expertise
• Strong communication skills
• Good leadership skills to guide and mentor the work of less experienced personnel
• Ability to be a high-impact player on multiple simultaneous engagements
• Ability to think strategically, balancing long and short-term priorities
• Willingness to travel

What you should expect in this role

• Client or office environment / may work remotely
• Occasional evening and weekend work