Lead Data Engineer

You are a highly skilled professional with over 15 years of experience and strong expertise in Python Programming, Pyspark queries, AWS, GIS, and Palantir Foundry. Your primary responsibilities will include developing and enhancing data-processing, orchestration, and monitoring using popular open-source software, AWS, and GitLab automation. You will collaborate closely with product and technology teams to design and validate the capabilities of the data platform. Additionally, you will be responsible for identifying, designing, and implementing process improvements, automating manual processes, optimizing usability, and redesigning for greater scalability. Your role will also involve providing technical support and usage guidance to the users of the platform's services. You will drive the creation and refinement of metrics, monitoring, and alerting mechanisms to ensure visibility into production services. To be successful in this position, you should have experience building and optimizing data pipelines in a distributed environment, working with cross-functional teams, and proficiency in a Linux environment. You must have at least 4 years of advanced working knowledge of SQL, Python, and PySpark queries. Knowledge of Palantir and experience with tools such as Git/Bitbucket, Jenkins/CodeBuild, and Code Pipeline is highly desirable. Additionally, experience with platform monitoring and alerting tools will be beneficial for this role.,