CIS Master's Thesis Defense by Ojonukpe Ameh

Wednesday, May 07, 2025 at 12:30pm to 1:30pm

Date & Time: Wednesday, May 7, 2025 - 12:30 pm - 1:30 pm 
Location: Dion 305 
Advisor: Dr. Yuchou Chang, Computer & Information Science, UMass Dartmouth

Committee Members: 
Dr. Firas Khatib, Computer & Information Science, UMass Dartmouth  
Dr. Adnan El-Nasan, Computer & Information Science, UMass Dartmouth

Abstract:

Contrastive Language-Image Pre-training (CLIP) Models refer to Machine Learning models that learn by contrasting between correct and incorrect pairings of text and image, with broad initialized foundations from training on large datasets. They explore an emergent space in Artificial Intelligence research as they offer a way to map multiple modalities of data together, and exhibit impressive zero-shot predictions. An important part of advancing the viability of CLIP-like models is thorough analysis to characterize as much as possible of their inner workings and increase explainability. Pursuing explainability in particularly multimodal models, where decision processes are quite opaque, is an important part of continued improvements. It provides easily intelligible interpretations of their outputs and decision-making processes, and also aids their widespread adoption. CLIP, the widely used foundational contrastive model, serves as a benchmark for multimodal representation learning and makes it the ideal choice for explainability-focused analysis. This thesis will introduce an approach for conducting perturbation-based analysis, investigating how systematic modifications to input images affect their similarity scores and semantic alignment, and providing structured and comprehensible visual and statistical outputs. One experiment identifies inputs highly susceptible to specific perturbations, indicating that the learned representation of those inputs is based on the unperturbed image features. Another experiment discussed visualizes semantic drift across a controlled set of text labels, illustrating how perturbations change the conceptual understanding of input.

All CIS Graduate Students are encouraged to attend.

For further questions please contact Dr. Yuchou Chang 

Dion