Data architect

As a Data Engineer, your primary responsibility will be to design and build production data pipelines within a hybrid big data architecture. You will use languages such as Scala, Python, and Talend to create efficient pipelines from data ingestion to consumption. Additionally, you will contribute to the entire data life-cycle, focusing on coding, debugging, and testing processes. Troubleshooting and optimizing ETL pipelines will also be a key aspect of your role, along with thorough documentation of each process. - Design and build production data pipelines using Scala, Python, and Talend within a hybrid big data architecture. - Gather and address technical and design requirements. - Refactor existing applications to optimize performance by setting appropriate architecture and integrating best practices. - Participate in the entire data life-cycle with a focus on coding, debugging, and testing. - Troubleshoot and debug ETL pipelines. - Document each process thoroughly. - BSc degree in Computer Science/Computer Engineering (Masters is a plus). - 2+ years of experience as a Data Engineer. - In-depth understanding of core ETL concepts, Data Modelling, Data Lineage, Data Governance, Data Catalog, etc. - 2+ years of work experience in Scala, Python, Java. - Good Knowledge of Big Data Tools such as Spark/HDFS/Hive/Flume, etc. - Hands-on experience with ETL tools like Talend/Informatica is a plus. - Good knowledge of Kafka and spark streaming is a big plus. - 2+ years of experience in using Azure cloud and its resources/services (Azure Data factory, Azure Databricks, SQL Synapse, Azure Devops, Logic Apps, Power Bi, Azure Event Hubs, etc). - Strong experience in Relational Databases (MySQL, SQL Server). - Exposure to data visualization tools like Power BI/Qlik Sense/MicroStrategy. - 2+ years of experience in developing APIs (REST & SOAP protocols). - Strong knowledge in Continuous Integration & Continuous Deployment (CI/CD) using Docker containers, Jenkins, etc. - Strong competencies in algorithms and software architecture. The job may also involve working with on-prem systems and an understanding of the entire web development process including design, development, and deployment. Experience in automated testing, including unit testing & UI testing, would be beneficial.,