DE&A - AIML - Data Science - Conventional AI - Search

Engineered a scalable data pipeline that seamlessly moved data from diverse sources, including REST APIs, on-premise databases, and SharePoint, into Snowflake using Azure Data Factory, significantly improving data integration efficiency. Led successful requirement gathering sessions, translating complex business needs into actionable technical specifications, which directly contributed to the projects successful delivery. Implemented robust CI/CD pipelines for Azure Data Factory components and Azure Databricks notebooks, enabling smooth and reliable deployments to higher environments using Azure DevOps. Automated the data ingestion process,eliminating 90% of manual data processing, drastically improving operational efficiency and reducing human error. Developed an enterprise-wide data model,consolidating data from multiple sources and ensuring consistent, organization-wide data access, leading to improved decision-making capabilities. Optimized data processing capabilities,allowing the system to handle 2GB of data every 2 hours with high performance and reliability, ensuring the infrastructure met business needs. Implemented parallelism in data pipelines,reducing pipeline execution time significantly, boosting performance, and enhancing data flow efficiency. Designed and documented a comprehensive architecture, detailing data transformation logic and Snowflake schema design, establishing a clear framework for future scalability and maintenance. Collaborated effectively with business users,ensuring data accessibility, gathering critical feedback, and providing ongoing support that enhanced user satisfaction and system usability.