Privacy Raven: Implementing a proof of concept for model inversion

Summary: A Trail of Bits intern implemented a proof-of-concept model inversion attack in PrivacyRaven, a Python tool for testing deep-learning frameworks against privacy attacks, based on the paper "Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment" by Ziqi Yang et al. The attack trains an inversion model on an auxiliary dataset of scraped images and the target classifier's confidence values, then reconstructs approximations of training data from prediction vectors; the proof of concept, run for 300 epochs with batch sizes of 100, produced a final MSE loss of 0.706 against an MNIST-trained classifier queried with EMNIST data.
Bottom line: A black-box classifier that returns confidence values can leak approximations of the private data it was trained on, and PrivacyRaven's pipeline first runs a model extraction attack to obtain a white-box substitute, meaning the barrier to attempting this is a query interface and scraped public images.

Read the original at Trail of Bits Blog ↗