Data Engineer – ADF/Databricks
About Company
Papigen is a fast-growing global technology services company, delivering innovative digital solutions through deep industry experience and cutting-edge expertise. We specialize in technology transformation, enterprise modernization, and dynamic areas like Cloud, Big Data, Java, React, DevOps, and more. Our client-centric approach combines consulting, engineering, and data science to help businesses evolve and scale efficiently.Scope of WorkCIT Operations – Knowledge & Learning Portfolio is seeking a highly skilled Data Engineer to support enterprise data engineering and search initiatives. The selected candidate will be responsible for the design, development, and optimization of data pipelines using Azure Data Factory (ADF), Databricks, and related Azure technologies. The role requires strong experience in ETL/ELT design, data governance, cloud security compliance, and cross-team collaboration in an Agile/SCRUM environment.Key Responsibilities
- Design and develop data pipelines for enterprise search solutions using Azure Data Factory and Databricks.
- Develop and document Low-Level Designs (LLD), Unit Testing Plans, and Unit Test Case documentation.
- Build ETL/ELT pipelines in ADF to extract data from on-premises, cloud, and third-party sources.
- Implement data cleansing, enrichment, and aggregation logic.
- Provide mentorship and technical guidance to junior team members, promoting best practices in cloud data engineering.
- Ensure compliance with security standards and best practices in cloud architecture and data handling, especially when dealing with sensitive information.
- Assist in Integration Testing, documentation, and preparation of artifacts required for OIS (Office of Information Security) clearance.
- Perform code optimization and ensure adherence to bank coding standards.
- Actively participate in SCRUM ceremonies and collaborate effectively with onsite and offshore teams.
- Ensure timely resolution of issues and conduct root cause analysis for recurring problems.
- Apply security compliance measures including application and data security assessments and remediation per WB OIS policies and standards.
- Proactively manage work queue, timelines, and deliverables, communicating trade-offs and dependencies as needed.
Required Skills & Experience
- Bachelor’s degree in Computer Science, Engineering, or equivalent relevant experience.
- 5+ years of experience in Data Engineering roles.
- Strong hands-on expertise in:
- Azure Data Factory (ADF), Databricks, Azure Data Lake
- Python programming and scripting
- ETL/ELT pipeline design & development
- Experience in cloud-native and hybrid solutions leveraging Azure technologies (e.g., ASE, caching solutions).
- Proven ability in logging, monitoring, performance tuning, and addressing production issues.
- Strong understanding of infrastructure design for private and public cloud.
- Ability to contribute to systems design within established frameworks.
- Familiarity with security standards and compliance requirements in cloud environments.
- Excellent skills in creating architectural documentation, design patterns, and technical specifications.
- Experience mentoring, guiding, and collaborating with technical teams across distributed geographies.
- Strong analytical, problem-solving, and communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.
- Organized and detail-oriented with strong ability to manage deliverables, documentation, and stakeholder communications.
Skills: agile,python scripting,databricks,scrum,performance tuning,etl,azure data factory,azure data lake,python automation
