Demonstrating and Characterizing Frog-Boiling Poisoning Against Drift-Aware Continual Learners
Microsoft Teams
Dr. Gokhan Kul
gkul@umassd.edu
https://teams.microsoft.com/meet/217648838909099?p=vkJbBE4Jvu6m4EWYJN
Thesis Advisor: Dr. Gokhan Kul - Computer & Information Science
Committee Members: Dr. Joshua Carberry - Computer & Information Science and Dr. Adnan El-Nasan - Computer & Information Science
Abstract: The implicit assumption of stationary data built into our framework of training machine learning systems has increasingly been found faulty. There are many domains where a model trained once and left to run in perpetuity loses classification accuracy over time as the data it encounters diverges from the specific character of the data used for its training. This phenomenon has a name, concept drift. There has been an expanding body of work to combat it, much of which relies on methods of continual learning, using the new data to update the model to adapt to the drift as it is encountered. This work has a fundamental tension: how do we adapt to the changing character of the data while also retaining the original fundamental understanding the model contains. With this thesis we aim to explore how this adaptation opens up a new attack vector in these systems, and how an adversary who can control a small fraction of the data stream can corrupt this adaptation process, crafting poison samples to slowly degrade the model's performance over time as well as aim to create a foundation to characterize the nature of this adversarial drift and how we can detect it. To this effect we demonstrate a white-box frog-boiling attack on an autoencoder that uses the Strategic Selection and Forgetting (SSF) framework as its drift adaptation mechanism. The model acts as a traditional intrusion detection system, trained to let benign, regular traffic through while flagging packets that constitute network attacks. SSF maintains a continually updated buffer of samples chosen to represent the current character of the data stream as faithfully as possible, and this buffer serves as the base of knowledge for continual retraining. The goal of the attack is to turn that adaptation mechanism against itself, expanding the model's learned representation of benign traffic outward round over round until it overlaps a chosen class of attack, so that attacks of that class pass as benign while the model's judgment of all other traffic is left largely untouched. Each round, the adversary submits poison the model still accepts as benign, drawn a step closer to the target class than the round before, so that the buffer when retrained on, induces a creep in the learned representation that marches steadily toward the attacker's goal. A straightforward interpolation between benign and attack samples is shown to induce this effect but somewhat inconsistently. Thus, to make a reliable attack we adapt feature collision with watermarking, a targeted clean-label poisoning technique, into a form that drives the boil consistently across seeds. Detecting this attack directly is difficult because no single sample betrays it. Each poisoning step is minute and arrives through the same adaptation the model applies to any drift. We find the attack only surfaces in the shape of the drift it leaves across many rounds. We characterize that drift against a synthetic benign-drift background and identify two signals that mark it as adversarial. A Webb input-space directness measure captures the sustained, directional path of a boil, setting it apart from the aimless wandering of natural drift, while a measure of the model’s contrastive loss catches the concentration of samples that don’t cleanly get folded into the benign region. Together these give early warning of a boil in progress before it has degraded the model's accuracy, laying a foundation for detecting this class of attack against continual learners.
For further information please contact Dr. Gokhan Kul at gkul@umassd.edu.