Data Lineage Developer

ABOUT OUR DEPARTMENT

Technology is the key differentiator that ensures that we manage our global businesses and serve clients on a market-leading platform that is resilient, safe, efficient, smart, fast and flexible. Technology redefines how we do business in global, complex and dynamic financial markets. We have a large number of award winning technology platforms that help to propel our Firms businesses to be the top in the market. We have built strong techno-functional teams which partner with our offices globally taking global ownership of systems and products. We have a vibrant and diverse mix of technologists working on different technologies and functional domains. There is a large focus on innovation, inclusion, giving back to the community and sharing knowledge.The TEDRA department maintains, distributes, and reports on Institutional Securities data. It is also responsible for developing and supporting applications for middle-office cash processing that facilitate straight-through processing by enriching trades and providing client allocation tools.Our team is looking for a talented and experienced professional to join and contribute to the collection and cataloging of the Metadata and Lineage of data persisted and distributed across the plant to the Firmwide Data Catalog. The ideal candidate will have a deep understanding of data architecture, data flow, data governance practices, and metadata cataloging. In this role, you will be responsible for designing, implementing, and maintaining a comprehensive data lineage solution that ensures visibility and traceability of data throughout its lifecycle across various systems and platforms. This is a challenging and dynamic role, requiring the data lineage developer to partner with other global technical and business teams inside and outside of the TEDRA area to ensure that data flows are well-documented, optimized, and auditable. Your work will directly contribute to our organization's ability to improve data quality, ensure regulatory compliance, and enhance decision-making through trusted data.

Responsibilities:

• Analysis of business processes to understand current setup, navigating through multiple systems, products and coordinate with multiple stakeholders.
• Work with stakeholders to define and map the end-to-end flow of data, identifying sources, transformations, destinations, and dependencies.
• Implement solutions to catalog metadata from diverse data sources, including MongoDB, Snowflake, and relational databases (e.g., SQL Server, DB2).
• Collaborate with firmwide data governance teams to ensure that lineage metadata is integrated with Collibra for auditing and compliance purposes.
• Develop processes to monitor data quality and ensure that data lineage tracks and reflects accurate data transformations and business rules.
• Design and implement automation tools and scripts to capture and maintain data lineage information and catalog updates on a scheduled basis.
• Create clear, understandable, and accurate documentation of data lineage for business users and technical teams.
• Identify and resolve issues related to data inconsistencies, lineage discrepancies, or errors in data transformation processes.
• Document technical requirements, specifications, and solutions.

Required skills:

• 4+ years of development experience.
• Experience/exposure in data engineering, data governance, or related fields, with a focus on data lineage and metadata cataloging.
• Strong expertise in data modeling, SQL, Stored Procedures/triggers, etc.
• Should possess good knowledge of OO, data structures and algorithms.
• Experience with data modeling, ETL pipelines, and data transformation.
• Experience with SQL and relational databases (e.g., Sybase, DB2, SQL Server).
• Proficiency in Python, Java, or other scripting languages for data integration and automation tasks.
• Strong analytical and troubleshooting skills with the ability to solve complex data flow problems.
• Excellent written and verbal communication skills, with the ability to explain technical concepts to non-technical stakeholders.
• Ability to work in a fast-paced, collaborative environment with cross-functional teams.

Good to have:

• Experience with Collibra
• Knowledge of metadata management practices, including the ability to extract and catalog metadata from complex data sources such as MongoDB, Snowflake, and relational databases
• Understanding of data privacy regulations (e.g., GDPR, CCPA) and their impact on data lineage and governance
• Experience of working in Agile teams