Big Data Engineer- Solution architect

Role & responsibilities

• Architectural Vision & Strategy:Define and articulate the technical vision, strategy, and roadmap for Big Data, data streaming, and NoSQL solutions, aligning with overall enterprise architecture and business goals.

• Solution Design & Blueprinting:Lead the end-to-end design of scalable, secure, and performant data architectures leveraging Apache Spark, Kafka, Hadoop Ecosystem components, Couchbase, and other relevant technologies. Create detailed architectural blueprints, including system diagrams, data flow diagrams, integration patterns, and non-functional requirements.

• Technology Evaluation & Selection:Research, evaluate, and recommend new technologies, tools, and platforms to address evolving business needs and technical challenges, conducting proof-of-concepts (PoCs) where necessary.

• Technical Leadership & Guidance:Provide expert technical leadership and mentorship to development teams, ensuring adherence to architectural principles, coding standards, and best practices. Conduct thorough code reviews to maintain high quality and optimize performance.

• Stakeholder Communication:Effectively communicate complex technical concepts, architectural decisions, risks, and trade-offs to diverse audiences, including technical teams, business stakeholders, and senior management.

• Performance & Scalability:Design solutions with a strong emphasis on performance, scalability, reliability, and cost-effectiveness. Identify and resolve performance bottlenecks and propose optimization strategies for existing and new systems.

• Data Governance & Security:Architect solutions that adhere to data governance policies, security standards, compliance regulations (e.g., GDPR, data residency), and privacy by design principles.

• Hands-on Contribution:Remain hands-on with technology by contributing to critical code modules, developing prototypes, and performing in-depth code reviews to ensure design integrity and efficiency.

• Documentation:Create and maintain comprehensive technical documentation, including architecture diagrams, design specifications, and implementationguidelines.

Required Skills & Experience:

• 15+ years of progressive experiencein software development, data engineering, and solution architecture roles, with a strong focus on large-scale distributed systems.

• Expertise in Big Data Technologies:

• Apache Spark:Deep expertise in Spark architecture, Spark SQL, Spark Streaming, performance tuning, and optimization techniques. Experience with data processing paradigms (batch and real-time).

• Hadoop Ecosystem:Strong understanding of HDFS, YARN, Hive, and other related Hadoop components.

• Real-time Data Streaming:

• Apache Kafka:Expert-level knowledge of Kafka architecture, topics, partitions, producers, consumers, Kafka Streams, KSQL, and best practices for high-throughput, low-latency data pipelines.

• NoSQL Databases:

• Couchbase:In-depth experience with Couchbase (or similar document/key-value NoSQL databases like MongoDB, Cassandra), including data modeling, indexing, querying (N1QL), replication, scaling, and operational best practices.

• API Design & Development:Extensive experience in designing and implementing robust, scalable, and secure APIs (RESTful, GraphQL) for data access and integration.

• Programming & Code Review:

• Hands-on coding proficiencyin at least one relevant language (Python, Scala, Java) with a preference for Python and/or Scala for data engineering tasks.

• Proven experience in leading and performing code reviews, ensuring code quality, performance, and adherence to architectural guidelines.

• Cloud Platforms:Extensive experience designing and implementing solutions on at least one major cloud platform (AWS, Azure, GCP), leveraging their Big Data, streaming, and compute services.

• Database Fundamentals:Solid understanding of relational database concepts, SQL, and data warehousing principles.

• System Design & Architecture Patterns:Deep knowledge of various architectural patterns (e.g., Microservices, Event-Driven Architecture, Lambda/Kappa Architecture, Data Mesh) and their application in data solutions.

DevOps & CI/CD:Familiarity with DevOps principles, CI/CD pipelines, infrastructure as code (IaC), and automated deployment strategies for data platforms.