Big Data Engineer

Notice Period-Immediate-10 days

The Data Engineer will lead end-to-end Big Data architecture design and build scalable data platforms that support high-volume, high-velocity data. The role emphasizes performance tuning of large-scale systems to deliver low-latency, cost-efficient data processing and analytics. The engineer will be hands-on with SQL and Python to design, implement, and optimize robust data pipelines. They will also strengthen data governance and security practices and explore AI-driven techniques to enhance data quality and processing efficiency.

Required Qualifications

• Big Data architecture design
• Performance tuning for large-scale systems
• Advanced SQL (query design, optimization)
• Python programming for data engineering
• Advanced data governance (lineage, access controls, cataloging) - preferred
• Data security (encryption, masking, auditing) - preferred
• AI-driven data processing (e.g., anomaly detection, entity resolution, automated quality) - preferred

Responsibilities

• Design and evolve Big Data architectures that support large-scale ingestion, storage, processing, and analytics.
• Own performance tuning across data pipelines and platforms, including query optimization and efficient data layouts.
• Develop and maintain robust, testable data pipelines using SQL and Python.
• Establish and improve data governance practices: lineage, access control, standards, and documentation.
• Implement and enforce data security controls, including encryption, masking, and auditing for compliance.
• Evaluate and apply AI-driven approaches to improve data processing, quality, and observability.
• Collaborate with stakeholders to translate business needs into scalable data designs and SLAs.
• Perform code reviews, establish best practices, and create technical documentation.