Data Engineer

Required Skills:

• Python
• Processing of large quantities of text documents
• Extraction of text from Office and PDF documents
• Input json to an API, output json to an API
• Nifi (or similar technology compatible)
• Basic understanding of AI/ML concepts
• Database/Search engine/SOLR skills
• SQL – build queries to analyze, create and update databases
• Understands the basics of hybrid search
• Experience working with terabytes (TB) of data
• Basic OpenML/Python/Azure knowledge
• Scripting knowledge/experience in an Azure environment to automate
• Cloud systems experience related to search and databases

Platforms:

• DataBricks
• Snowflake
• ESRI ArcGIS / SDE
• New GenAI app being developed

Scope of work

1. Ingest TB of data from multiple sources identified by the Ingestion Lead

2. Optimize data pipelines to improve on data processing, speed, and data availability

4. Make data available for end users from several hundred LAN and SharePoint areas

5. Monitor data pipelines daily and fix issues related to scripts, platforms, and ingestion

6. Work closely with the Ingestion Lead & Vendor on issues related to data ingestion

Technical Skills demonstrated:

1. SOLR - Backend database

2. Nifi - Data movement

3. Pyspark - Data Processing

4. Hive & Oozie - For jobs monitoring

5. Querying - SQL, HQl and SOLR querying

6. SQL

7. Python