AWS Databricks - Lead

• Technical Leadership: Demonstrate leadership, and ability to guide business and technology teams in adoption of best practices and standards
• Design & Development: Design, develop, and maintain robust, scalable, and high-performance data estate
• Architecture: Architect and design robust data solutions that meet business requirements & include scalability, performance, and security.
• Quality: Ensure the quality of deliverables through rigorous reviews, and adherence to standards.
• Agile Methodologies: Actively participate in agile processes, including planning, stand-ups, retrospectives, and backlog refinement.
• Collaboration: Work closely with system architects, data engineers, data scientists, data analysts, cloud engineers and other business stakeholders to determine optimal solution & architecture that is future-proof too.
• Innovation: Stay updated with the latest industry trends and technologies, and drive continuous improvement initiatives within the development team.
• Documentation: Create and maintain technical documentation, including design documents, and architectural user guides.

Technical Responsibilities:

• Optimize data pipelines for performance and efficiency.
• Work with Databricks clusters and configuration management tools.
• Use appropriate tools in the cloud data lake development and deployment. Developing/implementing cloud infrastructure to support current and future business needs.
• Provide technical expertise and ownership in the diagnosis and resolution of issues.
• Ensure all cloud solutions exhibit a higher level of cost efficiency, performance, security, scalability, and reliability.
• Manage cloud data lake development and deployment on AWSDatabricks.
• Manage and create workspaces, configure cloud resources, view usage data, and manage account identities, settings, and subscriptions in Databricks

Required Technical Skills:

• Experience & Proficiency with Databricks platform - Delta Lake storage, Spark (PySpark, Spark SQL). Must be well versed with Databricks Lakehouse, Unity Catalog concept and its implementation in enterprise environments.
• Familiarity of data design pattern - medallion architecture to organize data in a Lakehouse.
• Experience & Proficiency with AWS Data Services S3, Glue, Athena, Redshift etc.| Airflow scheduling
• Proficiency in SQL and experience with relational databases.
• Proficiency in at least one programming language (e.g., Python, Java) for data processing and scripting.
• Experience with DevOps practices - AWS DevOps for CI/CD, Terraform/CDK for infrastructure as code
• Good understanding of

  data principles, Cloud Data Lake design & development including data ingestion, data modeling and data distribution.

• Jira:

  Proficient in using Jira for managing projects and tracking progress.

Other Skills:

• Strong communication and interpersonal skills.
• Collaborate with data stewards, data owners, and IT teams for effective implementation
• Understanding of business processes and terminology preferably

  Logistics

• Experienced with Scrum and Agile Methodologies

Qualification

• Bachelors degree in information technology or a related field. Equivalent experience may be considered.
• Overall experience of

  8-12 years

  in Data Engineering