Over the approaching many years, safety dangers related to AI techniques can be a significant focus of researchers’ efforts. One of many least explored dangers at present is the opportunity of trojanizing an AI mannequin. This includes embedding hidden performance or intentional errors right into a machine studying system that seems to be working appropriately at first look. There are numerous strategies to create such a Computer virus, differing in complexity and scope — and so they should all be protected in opposition to.
Malicious code within the mannequin
Sure ML mannequin storage codecs can comprise executable code. For instance, arbitrary code could be executed whereas loading a file in a pickle format, the usual Python format used for information serialization (changing information right into a kind that’s handy for storing and transferring). Significantly, this format is utilized in a deep studying library PyTorch. In one other standard machine studying library, TensorFlow, fashions within the .keras and HDF5 codecs assist a “lambda layer”, which additionally executes arbitrary Python instructions. This code can simply conceal malicious performance.
TensorFlow’s documentation features a warning {that a} TensorFlow mannequin can learn and write information, ship and obtain community information, and even launch youngster processes. In different phrases, it’s basically a full-fledged program.
Malicious code can activate as quickly as an ML mannequin is loaded. In February 2024, roughly 100 fashions with malicious performance have been found within the standard repository of public fashions, Hugging Face. Of those, 20% created a reverse shell on the contaminated machine, and 10% launched extra software program.
Coaching dataset poisoning
Fashions could be trojanized on the coaching stage by manipulating the preliminary datasets. This course of, known as information poisoning, could be both focused or untargeted. Focused poisoning trains a mannequin to work incorrectly in particular circumstances (for instance, all the time claiming that Yuri Gagarin was the primary individual on the Moon). Untargeted poisoning goals to degrade the mannequin’s total high quality.
Focused assaults are tough to detect in a skilled mannequin as a result of they require very particular enter information. However poisoning the enter information for a big mannequin is expensive, because it requires altering a major quantity of information with out being detected.
In follow, there are identified circumstances of manipulating fashions that proceed to be taught whereas in operation. Essentially the most putting instance is the poisoning of Microsoft’s Tay chatbot, which was skilled to precise racist and extremist views in lower than a day. A extra sensible instance is the makes an attempt to poison Gmail’s spam classifier. Right here, attackers mark tens of hundreds of spam emails as official to permit extra spam via to consumer inboxes.
The identical aim could be achieved by altering coaching labels in annotated datasets or by injecting poisoned information into the fine-tuning technique of a pre-trained mannequin.
Shadow logic
A brand new methodology of maliciously modifying AI techniques is to introduce extra branches into the mannequin’s computational graph. This assault doesn’t contain executable code or tampering with the coaching course of, but the modified mannequin can exhibit a desired conduct in response to particular pre-determined enter information.
The assault leverages the truth that machine studying fashions use a computational graph to construction the computations required for his or her coaching and execution. The graph describes the sequence through which neural community blocks are linked and defines their operational parameters. Computational graphs are designed for every mannequin individually, though in some ML mannequin architectures they’re dynamic.
Researchers have demonstrated that the computational graph of an already skilled mannequin could be modified by including a department on the preliminary levels of its operation that detects a “particular sign” within the enter information; upon detection, the mannequin is directed to function underneath a individually programmed logic. In an instance from the research, the favored video object detection mannequin YOLO was modified to disregard folks in a body if a cup was additionally current.
The hazard of this methodology lies in its applicability to any fashions, no matter storage format, modality, or scope of utility. A backdoor could be applied for pure language processing, object detection, classification duties, and multimodal language fashions. Furthermore, such a modification could be preserved even when the mannequin undergoes additional coaching and fine-tuning.
shield AI fashions from backdoors
A key safety measure is the thorough management of the availability chain. This implies making certain that the origin of each part within the AI system is thought and freed from malicious modifications, together with:
- The code working the AI mannequin
- The computing surroundings through which the mannequin operates (normally cloud internet hosting)
- The information of the mannequin
- The information used for coaching
- The information used for fine-tuning
Main ML repositories are progressively implementing digital signatures to confirm fashions’ origins and code.
In circumstances the place strict management over the origins of information and code just isn’t possible, fashions from questionable sources needs to be averted in favor of respected suppliers’ choices.
It’s additionally essential to make use of safe codecs for storing ML fashions. Within the Hugging Face repository, warnings are displayed when loading fashions able to executing code; additionally, the first mannequin storage format is Safetensor, which blocks code execution.
