Data Architect

Responsibilities

Data Pipeline Development: Design, build, and maintain robust and scalable data pipelines using Databricks and Apache Spark, primarily with Scala, to ingest, process, and transform large datasets from various sources.ETL/ELT Implementation: Develop and optimize Extract, Transform, Load ETL or Extract, Load, Transform ELT processes within the Databricks environment to ensure data quality, consistency, and availability for analytical purposes.Experience on SnowflakeAPI Development: Design, develop, and deploy secure and efficient RESTful APIs using AWS services like Lambda and API Gateway to expose data or trigger data processing workflows.Cloud Infrastructure Management: Utilize AWS services for data storage, compute, and other infrastructure components related to data engineering solutions.Performance Optimization: Optimize Spark jobs and Databricks clusters for performance and cost efficiency.Data Governance and Security: Implement data governance best practices, ensuring data security and compliance within the Databricks and AWS environments, including securing API endpoints.Collaboration: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.Monitoring and Troubleshooting: Monitor data pipelines, identify and resolve issues, and ensure the reliability and availability of data systems.

Qualifications

Proficiency in Scala: Strong programming skills in Scala, specifically in the context of Apache Spark.Databricks Expertise: In-depth experience with Databricks platform, including Databricks notebooks, Delta Lake, and cluster management.Apache Spark Mastery: Extensive experience with Apache Spark for big data processing, including Spark SQL, Spark Streaming, and Spark MLlib.AWS Cloud Experience: Hands-on experience with AWS services, particularly Lambda, API Gateway, S3, EC2, and potentially other relevant services like Glue or Redshift.API Development: Proven ability to design and implement secure RESTful APIs, including authentication and authorization mechanisms.Data Warehousing Concepts: Understanding of data warehousing principles, data modeling, and schema design.SQL Proficiency: Strong SQL skills for data manipulation and analysis.Problem-Solving: Excellent analytical and problem-solving skills with a focus on delivering high-quality data solutions.Communication: Effective communication skills to collaborate with technical and non-technical stakeholders.