Job
Description
As a Senior Data Engineer at SMC Global Securities, you will play a crucial role in building and maintaining the data infrastructure that powers financial services and trading platforms. You will work with a diverse tech stack to handle massive volumes of real-time and historical financial data, ensuring high-quality, reliable information for analytics, research, and business teams. Your responsibilities will include: - Designing, building, and maintaining efficient and scalable real-time & batch data pipelines for financial data ingestion, processing, and analysis. - Owning the design and architecture of Data Lakehouse environments to align with business strategy and scalability requirements. - Implementing data modeling standards and various modeling techniques to create a robust data architecture for financial data. - Developing data pipelines using a Medallion Architecture for data quality, lineage, and auditability. - Optimizing data transformation processes for financial data using tools like DBT and Apache Spark. - Managing data storage solutions across Data Lake and Data Warehouse architectures. - Writing and optimizing SQL queries and Python scripts for data extraction, transformation, and loading. - Collaborating with different teams to understand data needs and build data models to support financial analysis and reporting. - Enforcing the implementation of a comprehensive Data Quality framework for data integrity and reliability. - Documenting data pipelines, data models, and processes for maintainability and knowledge sharing. Qualifications required for this role include: - Strong proficiency in SQL and Python with a focus on data manipulation and analysis. - Good understanding of data modeling concepts and various modeling techniques in a financial context. - Experience with Medallion and composable data architectures. - Hands-on experience with AWS cloud services like S3, Redshift, Athena, Glue, and Lambda. - Familiarity with open-source data tools such as Airflow, DBT, DLT, and Airbyte. - Knowledge of Hive & Iceberg tables. - Prior experience in building and maintaining data pipelines with large volumes of financial data. - Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience. - Preferred experience in the financial services, trading, or fintech industry. As a Senior Data Engineer at SMC Global Securities, you will play a crucial role in building and maintaining the data infrastructure that powers financial services and trading platforms. You will work with a diverse tech stack to handle massive volumes of real-time and historical financial data, ensuring high-quality, reliable information for analytics, research, and business teams. Your responsibilities will include: - Designing, building, and maintaining efficient and scalable real-time & batch data pipelines for financial data ingestion, processing, and analysis. - Owning the design and architecture of Data Lakehouse environments to align with business strategy and scalability requirements. - Implementing data modeling standards and various modeling techniques to create a robust data architecture for financial data. - Developing data pipelines using a Medallion Architecture for data quality, lineage, and auditability. - Optimizing data transformation processes for financial data using tools like DBT and Apache Spark. - Managing data storage solutions across Data Lake and Data Warehouse architectures. - Writing and optimizing SQL queries and Python scripts for data extraction, transformation, and loading. - Collaborating with different teams to understand data needs and build data models to support financial analysis and reporting. - Enforcing the implementation of a comprehensive Data Quality framework for data integrity and reliability. - Documenting data pipelines, data models, and processes for maintainability and knowledge sharing. Qualifications required for this role include: - Strong proficiency in SQL and Python with a focus on data manipulation and analysis. - Good understanding of data modeling concepts and various modeling techniques in a financial context. - Experience with Medallion and composable data architectures. - Hands-on experience with AWS cloud services like S3, Redshift, Athena, Glue, and Lambda. - Familiarity with open-source data tools such as Airflow, DBT, DLT, and Airbyte. - Knowledge of Hive & Iceberg tables. - Prior experience in building and maintaining data pipelines with large volumes of financial data. - Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience. - Preferred experience in the financial services, trading, or fintech industry.
