Data Engineer

Role: Data Engineer

Location: Noida

JD:

We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way.

Key Responsibilities:

• Data Pipeline Development:

  Design, build, and maintain robust, scalable, and efficient data pipelines to collect, process, and store large volumes of data from various sources.

• Data Integration:

  Integrate data from multiple sources, including APIs, databases, and external datasets, ensuring data consistency and reliability.

• Data Modelling:

  Develop and maintain data models and schemas that support efficient data storage, retrieval, and analytics.

• Database Management

  : Manage and optimize databases, ensuring their performance, availability, and security

• Data Quality:

  Implement and monitor data quality checks to ensure the accuracy, completeness, and consistency of data. Perform analysis required to troubleshoot data related issues and assist in the resolution of data issues.

• Automation:

  configure data extraction and load jobs and automate repetitive tasks and processes to improve efficiency and reduce errors. Improve performances of jobs and pipelines by applying optimisation and automation techniques.

• Documentation:

  Maintain clear and comprehensive documentation of data pipelines, data models, and data integration processes. Maintenance of time allocation reports.
• Develop understanding of business/ processes and high-level understanding of high-quality digital product delivery

• Ways of working:

  Follow Agile and SDLC processes including the creation of data related user stories. Perform code peer reviews and testing as per SDLC or industry standards.

• Collaboration:

  Work closely with internal customers, product owners, various core team members, and other stakeholders to understand their data needs and provide appropriate solutions. Technical collaboration and oversight with Vendors as required. Create SOWs and work packets with vendors as required.

Education:

• Bachelors or Masters degree in computer science, data science, software engineering, information systems or a similar field or equivalent experience.

Experience and skills:

• Over 5+ years’ experience as a Data Engineer or in a similar role.
• Experience with data pipeline, ETL and workflow management tools (e.g., Databricks, Data Factory).
• Proficiency in programming languages such as SQL, Python, R, or Scala.
• Strong experience with SQL and database management (e.g., MySQL, PostgreSQL, SQL Server).
• Excellent problem-solving and analytical skills.
• Strong understanding of data architecture, data modelling, and ETL processes.
• Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously.
• Strong communication and collaboration skills.
• Experience with best practice UX / UI design & development of Business Intelligence data visualisation / exploration tools (Power Bi, Qlik etc)
• Familiarity with SDLC and Agile ways of working.
• Familiarity with DevOps tools and practices (e.g. Jenkins, Azure DevOps, etc).
• Familiarity with Atlassian tools like Bitbucket, JIRA, Confluence.