Technical Manager/Architect - Fullstack Data Engineering

Engineering Manager & Technical Architect - Full Stack Data Integration

Position Overview

We are seeking a seasoned Engineering Manager and Technical Architect with 13-17 years of industry experience to lead our data integration and full-stack development initiatives. This role combines hands-on technical leadership with team management, requiring deep expertise in building scalable applications that seamlessly integrate with diverse data ecosystems.

This role offers the opportunity to work at the intersection of cutting-edge technology and team leadership, driving innovation in data-centric applications while building and mentoring high-performing engineering teams.

Key Responsibilities

Technical Leadership & Architecture

• Hands-on Full Stack Development: Design and implement end-to-end solutions using modern full-stack technologies while maintaining architectural oversight
• Data Integration Architecture: Architect robust solutions for integrating applications with multiple data sources including:
  • Traditional databases (SQL/NoSQL)
  • Real-time streaming platforms (Kafka, Kinesis, Pulsar)
  • Structured data sources (APIs, data warehouses, lakes)
  • Unstructured data sources (document stores, blob storage, content management systems)
• Cross-Application Communication: Design and implement secure, scalable communication patterns between distributed systems using REST APIs, GraphQL, message queues, and event-driven architectures

Data Engineering Excellence

• Data Pipeline Design: Architect and implement robust ETL/ELT pipelines ensuring data integrity, quality, and lineage tracking
• Apache Spark Mastery: Design and optimize large-scale data processing workflows using Apache Spark for:
  • Batch processing and analytics
  • Streaming data processing (Spark Streaming/Structured Streaming)
  • Machine learning pipelines (Spark MLlib)
  • Complex data transformations at scale
• Performance Optimization: Design systems with optimal performance characteristics while maintaining data consistency and reliability
• Data Governance: Implement data quality frameworks, monitoring, and alerting systems to ensure data integrity across the ecosystem

Infrastructure & DevOps

• Multi-Environment Deployment: Design and manage deployment strategies across:
  • Kubernetes clusters (on-premises and cloud)
  • Public cloud platforms (AWS, Azure, GCP)
  • Hybrid and multi-cloud environments
• Performance Engineering: Implement comprehensive monitoring, profiling, and optimization strategies to achieve optimal performance-to-cost ratios
• Cost Optimization: Continuously analyze and optimize infrastructure costs while maintaining performance SLAs

Team Management & Leadership

• Cross-Functional Team Leadership: Manage diverse teams including:
  • Data Engineers
  • Application/Software Engineers
  • QA Engineers
  • DevOps Engineers
• Project Management: Drive project execution through effective task assignment, progress tracking, and risk mitigation
• Strategic Planning: Develop alternative solutions (Plan B/C) for critical project deliverables and technical challenges
• Mentorship: Provide technical guidance and career development for team members

Required Qualifications

Experience & Background

• 13-17 years of progressive experience in software engineering and technical leadership roles
• 5+ years in engineering management positions with direct team leadership experience
• Proven track record in similar Engineering Manager + Architect roles in data-intensive environments

Technical Expertise

• Full Stack Proficiency: Expert-level skills in modern full-stack technologies (React/Angular/Vue, Node.js/Python/Java, microservices)
• Data Integration Mastery: Deep understanding of data integration patterns, API design, and inter-system communication protocols
• Database Technologies: Extensive experience with both SQL (PostgreSQL, MySQL, Oracle) and NoSQL (MongoDB, Cassandra, Redis) databases
• Apache Spark Expertise: Hands-on experience with:
  • Spark Core, SQL, Streaming, and MLlib
  • Performance tuning and optimization of Spark applications
  • Integration with distributed storage systems (HDFS, S3, Delta Lake)
  • Spark deployment on Kubernetes and cloud environments
• Streaming Technologies: Hands-on experience with real-time data processing using Kafka, Apache Storm, Spark Streaming, or similar
• Cloud & Infrastructure: Production experience with:
  • Kubernetes orchestration and management
  • Major cloud platforms (AWS/Azure/GCP)
  • Infrastructure-as-Code (Terraform, CloudFormation)
  • CI/CD pipelines and DevOps practices
  
  

Preferred Qualifications

• Open Source Contribution: Active participation in open-source projects and communities, especially Apache Spark ecosystem
• Industry Certifications: Cloud certifications (AWS/Azure/GCP Solutions Architect), Kubernetes certifications, or Databricks certifications
• Advanced Apache Spark: Experience with Spark optimization techniques, custom data sources, and extending Spark functionality
• Domain Expertise: Experience in specific industries (fintech, healthcare, e-commerce) with complex data requirements

What You'll Deliver

• Scalable, maintainable full-stack applications with seamless data integration capabilities
• High-performing Apache Spark applications processing large-scale data with optimal resource utilization
• High-performing, cost-optimized infrastructure deployments across multiple environments
• Well-functioning, motivated engineering teams delivering consistent results
• Robust data pipelines ensuring data integrity and optimal performance
• Strategic technical roadmaps with comprehensive risk mitigation plans

Years Of Exp

10 to 18 years

Mandatory Skills

Python, Spark, Data Integration, Fullstack,

Location

Bengaluru, Karnataka, India