Big Data Consultant

As a Senior Data Engineer, you will be responsible for developing and enhancing data-processing, orchestration, monitoring, and more using popular open-source software, AWS, and GitLab automation. Your role involves collaborating with product and technology teams to design and validate the capabilities of the data platform. Additionally, you will identify, design, and implement process improvements by automating manual processes, optimizing for usability, and re-designing for greater scalability. Providing technical support and usage guidance to the users of the platform's services will be part of your responsibilities. You will also drive the creation and refinement of metrics, monitoring, and alerting mechanisms to ensure visibility into production services. Key Responsibilities: - Develop and enhance data-processing, orchestration, monitoring, and more using popular open-source software, AWS, and GitLab automation - Collaborate with product and technology teams to design and validate the capabilities of the data platform - Identify, design, and implement process improvements by automating manual processes, optimizing for usability, and re-designing for greater scalability - Provide technical support and usage guidance to users of the platform's services - Drive the creation and refinement of metrics, monitoring, and alerting mechanisms for visibility into production services Qualifications: - Experience in building and optimizing data pipelines in a distributed environment - Experience supporting and working with cross-functional teams - Proficiency working in a Linux environment - 4+ years of advanced working knowledge of SQL, Python, and PySpark - Knowledge of Palantir - Experience using tools such as Git/Bitbucket, Jenkins/CodeBuild, and CodePipeline - Experience with platform monitoring and alerts tools Kindly note that the company details are not provided in the Job Description.,