Recovering Vulnerability Detection on Realistically-Distributed Code: Code Property Graphs, Graph Neural Networks, and a Disciplined Imbalance Recipe

Thursday, August 27, 2026 at 2:00pm to 3:00pm

Dion 303
Dr. Asif Kamal Turzo
aturzo@umassd.edu

Advisor: Dr. Asif Kamal Turzo
Committee Members: Dr. Haiping Xu and Dr. Adnan El-Nasan

Abstract: Prior deep-learning vulnerability detectors report F1 scores as high as 99% on benchmark datasets, yet collapse to the trivial all-safe classifier when evaluated on realistically-distributed code. On Real-Vul (Chakraborty et al., 2024), a corpus that simulates deployment-scale class imbalance of roughly 0.3% positive, every published model studied scores ROC-AUC near 0.50 and vulnerable-class F1 near zero. This thesis demonstrates that the collapse is not fundamental. We rebuild Real-Vul through a controlled Code Property Graph (CPG) pipeline, measure and remove a 37% content leak intrinsic to whole-codebase sampling, and train a relational graph neural network with a disciplined class-imbalance recipe: focal loss, class-aware undersampling, and fine-tuning of a GraphCodeBERT node encoder. On the deduplicated test set (n = 30,873; 518 vulnerable; 1.68% positive), the model reaches ROC-AUC 0.975 (95% CI [0.967, 0.983]) and vulnerable-class F1 0.773 (95% CI [0.746, 0.799]), where the strongest prior result on the same corpus reached F1 0.46 / AUC 0.82. Three controlled experiments locate the cause and the boundary of the result. An architectural ablation attributes the gain to the training recipe operating on a protocol-matched corpus; swapping the readout architecture leaves the metrics unchanged. A four-corpus transfer matrix shows that the result holds only within a single labeling protocol. A Java imbalance-reproduction experiment shows the recipe consistently beating no-handling on the same corpus while reaching a strong detector only where the corpus carries learnable structural signal at scale. The result is a working detector on realistic data, with an account of the conditions under which it works.

All CIS graduate students are encouraged to attend. For further information please contact Dr. Asif Kamal Turzo at aturzo@umassd.edu