Sr. Data Architect

DETAILED JOB DESCRIPTION

Roles & Responsibilities

• 17 to 22 years of experience in cloud-based systems and data platforms management with 8 to 10 years of experience as a Data Architect.


• Design, develop, and implement robust and scalable data architectures on Azure, considering performance, cost-efficiency, and security.


• Identify performance bottlenecks, analyse system behaviour under load, and develop strategies to improve efficiency and scalability.


• Provide guidance on system design and architecture to optimize performance, including database configurations, caching mechanisms, and code optimization.


• Develop and maintain scalable and efficient streaming frameworks.


• Integrate Salesforce solutions with other system components to meet business requirements.


• Utilize scripting languages such as Python for automation and data processing tasks.


• Collaborate with cross-functional teams, including development, operations, and security, to ensure effective cloud architecture and integration.


• Manage a team of 6 to 8 members with below skills.

Detailed Job Description

MUST HAVE

Azure Cloud Architecture (Expert Level - Required)

Advanced PySpark (Expert Level - Required)

- Mastery of DataFrame APIs and Spark SQL

- Performance tuning: broadcast joins, caching strategies, shuffle reduction

Databricks Expertise (Expert Level - Required)

- Cluster sizing, autoscaling, and spot instance optimization

- Job orchestration using Workflows with retries, alerts, and dependencies

PostgreSQL (Required)

- Query profiling and execution plan analysis

- Indexing strategies for analytical workloads

Performance & Scalability (Expert Level - Required)

- Designing horizontally scalable pipelines

- Spark memory management: executor sizing, GC tuning

GOOD TO HAVE

Kafka (Nice to have)

- Designing high-throughput Kafka consumers with Spark Structured Streaming

- Managing consumer groups and offset strategies

Structured Streaming in Spark (Nice to have)

- Stateful aggregations and windowed operations

- Exactly-once semantics with checkpointing and write-ahead logs

Machine Learning & MLOps (Nice to have)

- MLflow integration for model tracking and deployment

- Deploying Spark ML pipelines

Node.js - (Nice to have)

- Building REST APIs for data access and control

- Integrating with PostgreSQL