Job
Description
Role & responsibilities Lead the end-to-end implementation of a data cataloging solution within AWS (preferably AWS Glue Data Catalog or third-party tools like Apache Atlas, Alation, Collibra, etc.). Establish and manage metadata frameworks for structured and unstructured data assets in the data lake and data warehouse environments. Integrate the data catalog with AWS-based storage solutions such as S3, Redshift, Athena, Glue, and EMR. Collaborate with data Governance/BPRG/IT projects teams to define metadata standards, data classifications, and stewardship processes. Develop automation scripts for catalog ingestion, lineage tracking, and metadata updates using Python, Lambda, Pyspark or Glue/EMR customs jobs. Work closely with data engineers, data architects, and analysts to ensure metadata is accurate, relevant, and up to date. Implement role-based access controls and ensure compliance with data privacy and regulatory standards. Create detailed documentation and deliver training/workshops for internal stakeholders on using the data catalog. Preferred candidate profile AWS Certifications (e.g., AWS Certified Data Analytics, AWS Solutions Architect). Experience with data catalog tools like Alation, Collibra, or Informatica EDC. Or open sources tools hand-on experience. Exposure to data quality frameworks and stewardship practices. Knowledge of data migration with data catalog and data-mart is plus. 4 to 8+ years of experience in data engineering or metadata management roles. Proven expertise in implementing and managing data catalog solutions wiithin AWS environments. Strong knowledge of AWS Glue, S3, Athena, Redshift, EMR, Data Catalog and Lake Formation. Hands-on experience with metadata ingestion, data lineage, and classification processes. Proficiency in Python, SQL, and automation scripting for metadata pipelines. Familiarity with data governance and compliance standards (e.g., GDPR, RBI guidelines). Experience integrating with BI tools (e.g., Tableau, Power BI) and third-party catalog tools is a plus. Strong communication, Problem-solving, and stakeholder management skills.
