Lead Data Engineer

Responsibilities

• Responsible for the design, deployment, configuration, and operations of a multi-node big data cluster.
• This includes working with open source and/or commercial stacks to support the full SDLC.
• Resource will work to deploy, manage, and maintain development, test and production environments for the big data platform.
• Develop scripts to automate and streamline operations and configurations in the infrastructure.
• Specify, design, build, and support BI solutions by working closely with datalake team.
• Create dashboards and KPIs to show the business performance to management.
• Design and maintain data models used for reporting and analytics
• Work to identify infrastructure needs and provide support to developers and business users.
• Research performance issues.
• Optimize platform for performance.
• Troubleshoot and resolve issues in all operational environments.
• Work with a cross-functional team delivering software deployments.
• Forward thinking by continuously adopting new ideas and technologies to solve business problems.
• Own the design and development of automated solutions for recurring reporting and in-depth analysis.
• A problem solver and critical thinker.
  

Requirements

• Strong Experience in Data lake - Spark, distributed file system, Yarn, Cloud services (preferably GCP / AWS).
• Strong Experience in SQL tools - Vertica, Dremio product or any big data SQL.
• Scripting Knowledge - Shell, Python.
• Experience with ETL and OLAP concepts in building highly scalable data pipelines.
• Exposure to any Visualization systems is a plus (like Apache Superset, Tableau).
• Experience with Agile, data structures, data analysis and wrangling tools and technologies.
• Familiar with version control and relational databases.
• Strong Experience in monitoring, debugging and troubleshooting of services.
• Experience in providing on-call support.
  

Basic Qualifications

• Eucloid Data Solutions Awarded for the most promising Data Solutions Provider by Entrepreneur Magazine.
• Bachelor's / Master's Degree in Computer or related field from a reputed institution.
• 5+ years of professional experience in software, with most of them from a product company.
  

Preferred Qualifications

• Proficient in one or more technologies, such as AWS, EMR, Hadoop, Spark, SQL, Python, and Data Structures.
• Experience working in Linux based environment.
• Good communication and design skills.