Data Engineer

Job Summary:

Data Engineer will be responsible to Develop set processes for data mining, data modeling, and data production. Develop and maintain scalable data pipelines and builds out new integrations to support increases in data volume and complexity. Employ an array of technological languages and tools to connect data sets. Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.

Role & responsibilities

Data Engineer, Key Deliverables

• Work in collaboration of IT infrastructure, Business teams and team members to create/develop and maintains optimal data pipeline architecture, assemble large and complex data sets from various business sources (like ERP systems)
• Re-engineer SQL & spark code to make it performance optimised.
• Work with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
• Create Integration documents.
• Perform data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
• Define data assets (data models) and methodologies to populate data models.
• Design data integrations and data quality framework.
• Construct, install, test and maintain data management systems.
• Build high-performance algorithms and prototypes.
• Integrate up-and-coming data management and software engineering technologies into existing data structures.
• Research new uses for existing data

Basic Qualifications

Bachelor's/masters in computer science, Information Management/Science, or comparable professional area

Knowledge/Skills/Abilities:

• 2-3 years of programming experience in one or more languages like JAVA, python, scala etc.
• Conceptual understanding of distributed computing principles and frameworks like Hadoop, MapReduce etc.
• Good understanding and practical exposure to distribution processing frameworks like Apache Spark
• Expertise in SQL and Data warehousing concepts.
• Working SQL knowledge and experience working with relational databases
• Nice to Have: Exposure to any ETL tool and Azure cloud services.