Data Architect

• Assess and balance the current in-house AI platform architecture with long-term modernization goals.
• Lead the migration of existing Apache Spark workloads to Databricks, ensuring scalability and performance.
• Design and optimize end-to-end data pipelines for both batch and real-time streaming use cases.
• Implement robust orchestration using Apache Airflow (preferably cloud-native).
• Provide architectural guidance to integrate legacy and modern data components seamlessly.
• Drive data governance, data quality, and security as core architectural pillars.
• Collaborate with data engineering, analytics, and DevOps teams to deliver scalable solutions.

Required Skills

• 12+ years in data engineering or data architecture roles, with focus on modern data platforms.
• Strong hands-on experience with Apache Spark (on-prem and Databricks environments).
• Proven experience with Databricks on Azure or AWS.
• Solid expertise in Apache Airflow for building and managing distributed data workflows.
• Strong knowledge of structured streaming frameworks (e.g., Kafka, Spark Streaming).
• Sound understanding of cloud-native data architectures (Azure, AWS, or GCP).
• Excellent problem-solving and technical leadership skills.
• Preferred Qualifications
• Experience transitioning from in house data platforms to Databricks or cloud-native environments.
• Hands-on experience with Delta Lake, Unity Catalog, and performance tuning in Databricks.
• Expertise in Apache Airflow DAG design, dynamic workflows, and production troubleshooting.
• Certification in Databricks Data Engineering is highly desirable.
• Strong background in data modeling, Lakehouse architecture, and implementation patterns.
• Familiarity with telecom domain data is required.
• Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices.
• Exposure to AI/ML model integration within real-time or batch data pipelines .