[ { "path": ".gitignore", "content": "\n# Created by https://www.gitignore.io/api/linux,macos,python,pycharm\n# Edit at https://www.gitignore.io/?templates=linux,macos,python,pycharm\n\n### Linux ###\n*~\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n.fuse_hidden*\n\n# KDE directory preferences\n.directory\n\n# Linux trash folder which might appear on any partition or disk\n.Trash-*\n\n# .nfs files are created when an open file is removed but is still being accessed\n.nfs*\n\n### macOS ###\n# General\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# Icon must end with two \\r\nIcon\n\n# Thumbnails\n._*\n\n# Files that might appear in the root of a volume\n.DocumentRevisions-V100\n.fseventsd\n.Spotlight-V100\n.TemporaryItems\n.Trashes\n.VolumeIcon.icns\n.com.apple.timemachine.donotpresent\n\n# Directories potentially created on remote AFP share\n.AppleDB\n.AppleDesktop\nNetwork Trash Folder\nTemporary Items\n.apdisk\n\n### PyCharm ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/**/usage.statistics.xml\n.idea/**/dictionaries\n.idea/**/shelf\n\n# Generated files\n.idea/**/contentModel.xml\n\n# Sensitive or high-churn files\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n.idea/**/dbnavigator.xml\n\n# Gradle\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn. Uncomment if using\n# auto-import.\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n\n# CMake\ncmake-build-*/\n\n# Mongo Explorer plugin\n.idea/**/mongoSettings.xml\n\n# File-based project format\n*.iws\n\n# IntelliJ\nout/\n\n# mpeltonen/sbt-idea plugin\n.idea_modules/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\ncom_crashlytics_export_strings.xml\ncrashlytics.properties\ncrashlytics-build.properties\nfabric.properties\n\n# Editor-based Rest Client\n.idea/httpRequests\n\n# Android studio 3.1+ serialized cache file\n.idea/caches/build_file_checksums.ser\n\n# JetBrains templates\n**___jb_tmp___\n\n### PyCharm Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n.idea/sonarlint\n\n### Python ###\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don’t work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# End of https://www.gitignore.io/api/linux,macos,python,pycharm\n\n\n\n# Just to make sure pycharm does not ruin anything: \n.idea\n\n# To avoid uploading datasets to the github\ndatasets\n\n\n# To avoid uploading models\nmodels\n\n\n# SWP vim \n.*.swp\n" }, { "path": "README.md", "content": "# Free Adversarial Training \nThis repository belongs to the [Free Adversarial Training](https://arxiv.org/abs/1904.12843 \"Free Adversarial Training\") paper.\nThe implementation is inspired by [CIFAR10 Adversarial Example Challenge](https://github.com/MadryLab/cifar10_challenge \"Madry's CIFAR10 Challenge\") so to them we give the credit.\nThis repo is for the CIFAR-10 and CIFAR-100 datasets and is in Tensorflow. Our Free-m models can acheive comparable performance with conventional PGD adversarial training at a fraction of the time. \n\n**__News!__**: We have released our [ImageNet implementation of Free adversarial training in Pytorch](https://github.com/mahyarnajibi/FreeAdversarialTraining) !\n\n\n###### CIFAR-10 WRN 32-10 (L-inf epsilon=8):\n\n| Model | Natural | PGD-100 | CW-100 | 10 restart PGD-20 | train-time (min) |\n| --- | --- | --- | --- | --- | --- |\n| Natrual | 95.01 | 0.00 | 0.00| 0.00 | 780 |\n| Free-2 | 91.45 | 33.20 | 34.57 | 33.41 | 816 |\n| Free-4 | 87.83 | 40.35 | 41.96 | 40.73 | 800 |\n| **Free-8** | **85.96** | **46.19** | **46.60** | **46.33** | **785** |\n| Free-10 |83.94 | 45.79 | 45.86 | 45.94 | 785 |\n|Madry 7-PGD (public model) | 87.25 | 45.29 | 46.52 | 45.53 | 5418 |\n\n###### CIFAR-100 WRN 32-10 (L-inf epsilon=8):\n| Model | Natural | PGD-20 | PGD-100 | train-time (min) |\n| --- | --- | --- | --- | --- |\n| Natrual | 78.84 | 0.00 | 0.00 | 811 |\n| Free-2 | 69.20 | 15.37 | 14.86 | 816 |\n| Free-4 | 65.28 | 20.64 | 20.15 | 767 |\n| **Free-8** | **62.13** | **25.88** | **25.58** | **780** |\n| Free-10 | 59.27 | 25.15 | 24.88 | 776 |\n| Madry 2-PGD trained | 67.94 | 17.08 | 16.50 | 2053 |\n| Madry 7-PGD trained | 59.87 | 22.76 | 22.52 | 5157 |\n\n\n## Demo\nTo train a new robust model for free! run the following command specifying the replay parameter `m`:\n\n```bash\npython free_train.py -m 8\n```\n\nTo evaluate a robust model using PGD-20 with 2 random restarts run:\n\n```bash\npython multi_restart_pgd_attack.py --model_dir $MODEL_DIR --num_restarts 2\n```\nNote that if you have trained a CIFAR-100 model, even for evaluation, you should pass the dataset argument. For example:\n```bash\npython multi_restart_pgd_attack.py --model_dir $MODEL_DIR_TO_CIFAR100 --num_restarts 2 -d cifar100\n```\n\n## Requirements \nTo install all the requirements plus tensorflow for multi-gpus run: (Inspired By [Illarion ikhlestov](https://github.com/ikhlestov/vision_networks \"Densenet Implementation\") ) \n\n```bash\npip install -r requirements/gpu.txt\n```\n\nAlternatively, to install the requirements plus tensorflow for cpu run: \n```bash\npip install -r requirements/cpu.txt\n```\n\nTo prepare the data, please see [Datasets section](https://github.com/ashafahi/free_adv_train/tree/master/datasets \"Dataset readme\").\n\nIf you find the paper or the code useful for your study, please consider citing the free training paper:\n```bash\n@article{shafahi2019adversarial,\n title={Adversarial Training for Free!},\n author={Shafahi, Ali and Najibi, Mahyar and Ghiasi, Amin and Xu, Zheng and Dickerson, John and Studer, Christoph and Davis, Larry S and Taylor, Gavin and Goldstein, Tom},\n journal={arXiv preprint arXiv:1904.12843},\n year={2019}\n}\n```\n" }, { "path": "cifar100_input.py", "content": "\"\"\"\nUtilities for importing the CIFAR100 dataset.\n\nEach image in the dataset is a numpy array of shape (32, 32, 3), with the values\nbeing unsigned integers (i.e., in the range 0,1,...,255).\n\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport os\nimport pickle\nimport sys\nimport tensorflow as tf\nfrom tensorflow.examples.tutorials.mnist import input_data\nversion = sys.version_info\n\nimport numpy as np\n\nclass CIFAR100Data(object):\n \"\"\"\n Unpickles the CIFAR100 dataset from a specified folder containing a pickled\n version following the format of Krizhevsky which can be found\n [here](https://www.cs.toronto.edu/~kriz/cifar.html).\n\n Inputs to constructor\n =====================\n\n - path: path to the pickled dataset. The training data must be pickled\n into one file name train containing 50,000, and 10,000\n examples each, the test data\n must be pickled into a single file called test containing 10,000\n examples, and the 100 fine or 20 coarse class names must be\n pickled into a file called meta. The pickled examples should\n be stored as a tuple of two objects: an array of 50,000 32x32x3-shaped\n arrays, and an array of their 50,000 true labels.\n\n \"\"\"\n def __init__(self, path):\n #train_filenames = ['data_batch_{}'.format(ii + 1) for ii in range(5)]\n train_filename = 'train'\n eval_filename = 'test'#'test_batch'\n metadata_filename = 'meta'#'batches.meta'\n\n train_images = np.zeros((50000, 32, 32, 3), dtype='uint8')\n train_labels = np.zeros(50000, dtype='int32')\n #for ii, fname in enumerate(train_filenames):\n # cur_images, cur_labels = self._load_datafile(os.path.join(path, fname))\n # train_images[ii * 10000 : (ii+1) * 10000, ...] = cur_images\n # train_labels[ii * 10000 : (ii+1) * 10000, ...] = cur_labels\n train_images, train_labels = self._load_datafile(\n os.path.join(path, train_filename))\n eval_images, eval_labels = self._load_datafile(\n os.path.join(path, eval_filename))\n\n with open(os.path.join(path, metadata_filename), 'rb') as fo:\n if version.major == 3:\n data_dict = pickle.load(fo, encoding='bytes')\n else:\n data_dict = pickle.load(fo)\n\n #self.label_names = data_dict[b'label_names']\n self.label_names = data_dict[b'fine_label_names']\n for ii in range(len(self.label_names)):\n self.label_names[ii] = self.label_names[ii].decode('utf-8')\n\n self.train_data = DataSubset(train_images, train_labels)\n self.eval_data = DataSubset(eval_images, eval_labels)\n\n @staticmethod\n def _load_datafile(filename):\n with open(filename, 'rb') as fo:\n if version.major == 3:\n data_dict = pickle.load(fo, encoding='bytes')\n else:\n data_dict = pickle.load(fo)\n\n assert data_dict[b'data'].dtype == np.uint8\n image_data = data_dict[b'data']\n #image_data = image_data.reshape((10000, 3, 32, 32)).transpose(0, 2, 3, 1)\n image_data = image_data.reshape((-1, 3, 32, 32)).transpose(0, 2, 3, 1)\n #return image_data, np.array(data_dict[b'labels'])\n return image_data, np.array(data_dict[b'fine_labels'])\n\nclass AugmentedCIFAR100Data(object):\n \"\"\"\n Data augmentation wrapper over a loaded dataset.\n\n Inputs to constructor\n =====================\n - raw_cifar10data: the loaded CIFAR100 dataset, via the CIFAR100Data class\n - sess: current tensorflow session\n - model: current model (needed for input tensor)\n \"\"\"\n def __init__(self, raw_cifar100data, sess, model):\n assert isinstance(raw_cifar100data, CIFAR100Data)\n self.image_size = 32\n\n # create augmentation computational graph\n self.x_input_placeholder = tf.placeholder(tf.float32, shape=[None, 32, 32, 3])\n padded = tf.map_fn(lambda img: tf.image.resize_image_with_crop_or_pad(\n img, self.image_size + 4, self.image_size + 4),\n self.x_input_placeholder)\n cropped = tf.map_fn(lambda img: tf.random_crop(img, [self.image_size,\n self.image_size,\n 3]), padded)\n flipped = tf.map_fn(lambda img: tf.image.random_flip_left_right(img), cropped)\n self.augmented = flipped\n\n self.train_data = AugmentedDataSubset(raw_cifar100data.train_data, sess,\n self.x_input_placeholder,\n self.augmented)\n self.eval_data = AugmentedDataSubset(raw_cifar100data.eval_data, sess,\n self.x_input_placeholder,\n self.augmented)\n self.label_names = raw_cifar100data.label_names\n\n\nclass DataSubset(object):\n def __init__(self, xs, ys):\n self.xs = xs\n self.n = xs.shape[0]\n self.ys = ys\n self.batch_start = 0\n self.cur_order = np.random.permutation(self.n)\n\n def get_next_batch(self, batch_size, multiple_passes=False, reshuffle_after_pass=True):\n if self.n < batch_size:\n raise ValueError('Batch size can be at most the dataset size')\n if not multiple_passes:\n actual_batch_size = min(batch_size, self.n - self.batch_start)\n if actual_batch_size <= 0:\n raise ValueError('Pass through the dataset is complete.')\n batch_end = self.batch_start + actual_batch_size\n batch_xs = self.xs[self.cur_order[self.batch_start : batch_end], ...]\n batch_ys = self.ys[self.cur_order[self.batch_start : batch_end], ...]\n self.batch_start += actual_batch_size\n return batch_xs, batch_ys\n actual_batch_size = min(batch_size, self.n - self.batch_start)\n if actual_batch_size < batch_size:\n if reshuffle_after_pass:\n self.cur_order = np.random.permutation(self.n)\n self.batch_start = 0\n batch_end = self.batch_start + batch_size\n batch_xs = self.xs[self.cur_order[self.batch_start : batch_end], ...]\n batch_ys = self.ys[self.cur_order[self.batch_start : batch_end], ...]\n self.batch_start += batch_size\n return batch_xs, batch_ys\n\n\nclass AugmentedDataSubset(object):\n def __init__(self, raw_datasubset, sess, x_input_placeholder,\n augmented):\n self.sess = sess\n self.raw_datasubset = raw_datasubset\n self.x_input_placeholder = x_input_placeholder\n self.augmented = augmented\n\n def get_next_batch(self, batch_size, multiple_passes=False, reshuffle_after_pass=True):\n raw_batch = self.raw_datasubset.get_next_batch(batch_size, multiple_passes,\n reshuffle_after_pass)\n images = raw_batch[0].astype(np.float32)\n return self.sess.run(self.augmented, feed_dict={self.x_input_placeholder:\n raw_batch[0]}), raw_batch[1]\n\n" }, { "path": "cifar10_input.py", "content": "\"\"\"\nUtilities for importing the CIFAR10 dataset.\nEach image in the dataset is a numpy array of shape (32, 32, 3), with the values\nbeing unsigned integers (i.e., in the range 0,1,...,255).\n\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport os\nimport pickle\nimport sys\nimport tensorflow as tf\nfrom tensorflow.examples.tutorials.mnist import input_data\nimport numpy as np\nimport re\n\nversion = sys.version_info\n\n\nclass CIFAR10Data(object):\n \"\"\"\n Unpickles the CIFAR10 dataset from a specified folder containing a pickled\n version following the format of Krizhevsky which can be found\n [here](https://www.cs.toronto.edu/~kriz/cifar.html).\n Inputs to constructor\n =====================\n - path: path to the pickled dataset. The training data must be pickled\n into five files named data_batch_i for i = 1, ..., 5, containing 10,000\n examples each, the test data\n must be pickled into a single file called test_batch containing 10,000\n examples, and the 10 class names must be\n pickled into a file called batches.meta. The pickled examples should\n be stored as a tuple of two objects: an array of 10,000 32x32x3-shaped\n arrays, and an array of their 10,000 true labels.\n \"\"\"\n\n def __init__(self, path):\n path = CIFAR10Data.rec_search(path)\n train_filenames = ['data_batch_{}'.format(ii + 1) for ii in range(5)]\n eval_filename = 'test_batch'\n metadata_filename = 'batches.meta'\n\n train_images = np.zeros((50000, 32, 32, 3), dtype='uint8')\n train_labels = np.zeros(50000, dtype='int32')\n for ii, fname in enumerate(train_filenames):\n cur_images, cur_labels = self._load_datafile(os.path.join(path, fname))\n train_images[ii * 10000: (ii + 1) * 10000, ...] = cur_images\n train_labels[ii * 10000: (ii + 1) * 10000, ...] = cur_labels\n eval_images, eval_labels = self._load_datafile(\n os.path.join(path, eval_filename))\n\n with open(os.path.join(path, metadata_filename), 'rb') as fo:\n if version.major == 3:\n data_dict = pickle.load(fo, encoding='bytes')\n else:\n data_dict = pickle.load(fo)\n\n self.label_names = data_dict[b'label_names']\n for ii in range(len(self.label_names)):\n self.label_names[ii] = self.label_names[ii].decode('utf-8')\n\n self.train_data = DataSubset(train_images, train_labels)\n self.eval_data = DataSubset(eval_images, eval_labels)\n\n @staticmethod\n def rec_search(original_path):\n rx = re.compile(r'data_batch_[0-9]+')\n r = []\n for path, _, file_names in os.walk(original_path):\n r.extend([os.path.join(path, x) for x in file_names if rx.search(x)])\n if len(r) is 0: # TODO: Is this the best way?\n return original_path\n return os.path.dirname(r[0])\n\n @staticmethod\n def _load_datafile(filename):\n with open(filename, 'rb') as fo:\n if version.major == 3:\n data_dict = pickle.load(fo, encoding='bytes')\n else:\n data_dict = pickle.load(fo)\n\n assert data_dict[b'data'].dtype == np.uint8\n image_data = data_dict[b'data']\n image_data = image_data.reshape((10000, 3, 32, 32)).transpose(0, 2, 3, 1)\n return image_data, np.array(data_dict[b'labels'])\n\n\nclass AugmentedCIFAR10Data(object):\n \"\"\"\n Data augmentation wrapper over a loaded dataset.\n Inputs to constructor\n =====================\n - raw_cifar10data: the loaded CIFAR10 dataset, via the CIFAR10Data class\n - sess: current tensorflow session\n - model: current model (needed for input tensor)\n \"\"\"\n\n def __init__(self, raw_cifar10data, sess, model):\n assert isinstance(raw_cifar10data, CIFAR10Data)\n self.image_size = 32\n\n # create augmentation computational graph\n self.x_input_placeholder = tf.placeholder(tf.float32, shape=[None, 32, 32, 3])\n padded = tf.map_fn(lambda img: tf.image.resize_image_with_crop_or_pad(\n img, self.image_size + 4, self.image_size + 4),\n self.x_input_placeholder)\n cropped = tf.map_fn(lambda img: tf.random_crop(img, [self.image_size,\n self.image_size,\n 3]), padded)\n flipped = tf.map_fn(lambda img: tf.image.random_flip_left_right(img), cropped)\n self.augmented = flipped\n\n self.train_data = AugmentedDataSubset(raw_cifar10data.train_data, sess,\n self.x_input_placeholder,\n self.augmented)\n self.eval_data = AugmentedDataSubset(raw_cifar10data.eval_data, sess,\n self.x_input_placeholder,\n self.augmented)\n self.label_names = raw_cifar10data.label_names\n\n\nclass DataSubset(object):\n def __init__(self, xs, ys):\n self.xs = xs\n self.n = xs.shape[0]\n self.ys = ys\n self.batch_start = 0\n self.cur_order = np.random.permutation(self.n)\n\n def get_next_batch(self, batch_size, multiple_passes=False, reshuffle_after_pass=True):\n if self.n < batch_size:\n raise ValueError('Batch size can be at most the dataset size')\n if not multiple_passes:\n actual_batch_size = min(batch_size, self.n - self.batch_start)\n if actual_batch_size <= 0:\n raise ValueError('Pass through the dataset is complete.')\n batch_end = self.batch_start + actual_batch_size\n batch_xs = self.xs[self.cur_order[self.batch_start: batch_end], ...]\n batch_ys = self.ys[self.cur_order[self.batch_start: batch_end], ...]\n self.batch_start += actual_batch_size\n return batch_xs, batch_ys\n actual_batch_size = min(batch_size, self.n - self.batch_start)\n if actual_batch_size < batch_size:\n if reshuffle_after_pass:\n self.cur_order = np.random.permutation(self.n)\n self.batch_start = 0\n batch_end = self.batch_start + batch_size\n batch_xs = self.xs[self.cur_order[self.batch_start: batch_end], ...]\n batch_ys = self.ys[self.cur_order[self.batch_start: batch_end], ...]\n self.batch_start += actual_batch_size\n return batch_xs, batch_ys\n\n\nclass AugmentedDataSubset(object):\n def __init__(self, raw_datasubset, sess, x_input_placeholder,\n augmented):\n self.sess = sess\n self.raw_datasubset = raw_datasubset\n self.x_input_placeholder = x_input_placeholder\n self.augmented = augmented\n\n def get_next_batch(self, batch_size, multiple_passes=False, reshuffle_after_pass=True):\n raw_batch = self.raw_datasubset.get_next_batch(batch_size, multiple_passes,\n reshuffle_after_pass)\n images = raw_batch[0].astype(np.float32)\n return self.sess.run(self.augmented, feed_dict={self.x_input_placeholder:\n raw_batch[0]}), raw_batch[1]\n" }, { "path": "config.py", "content": "import configargparse\nimport pdb\n\ndef pair(arg):\n return [float(x) for x in arg.split(',')]\n\ndef get_args():\n parser = configargparse.ArgParser(default_config_files=[])\n parser.add(\"--config\", type=str, is_config_file=True, help=\"You can store all the config args in a config file and pass the path here\")\n parser.add(\"--model_dir\", type=str, default=\"models/model\", help=\"Path to save/load the checkpoints, default=models/model\")\n parser.add(\"--data_dir\", type=str, default=\"datasets/\", help=\"Path to load datasets from, default=datasets\")\n parser.add(\"--dataset\", \"-d\", type=str, default=\"cifar10\", choices=[\"cifar10\", \"cifar100\"], help=\"Path to load dataset, default=cifar10\")\n parser.add(\"--tf_seed\", type=int, default=451760341, help=\"Random seed for initializing tensor-flow variables to rule out the effect of randomness in experiments, default=45160341\") \n parser.add(\"--np_seed\", type=int, default=216105420, help=\"Random seed for initializing numpy variables to rule out the effect of randomness in experiments, default=216105420\") \n parser.add(\"--train_steps\", type=int, default=80000, help=\"Maximum number of training steps, default=80000\")\n parser.add(\"--out_steps\", \"-o\", type=int, default=100, help=\"Number of output steps, default=100\")\n parser.add(\"--summary_steps\", type=int, default=500, help=\"Number of summary steps, default=500\") \n parser.add(\"--checkpoint_steps\", \"-c\", type=int, default=1000, help=\"Number of checkpoint steps, default=1000\")\n parser.add(\"--train_batch_size\", \"-b\", type=int, default=128, help=\"The training batch size, default=128\")\n parser.add(\"--step_size_schedule\", nargs='+', type=pair, default=[[0, 0.1], [40000, 0.01], [60000, 0.001]], help=\"The step size scheduling, default=[[0, 0.1], [40000, 0.01], [60000, 0.001]], use like: --stepsize 0,0.1 40000,0.01 60000,0.001\") \n parser.add(\"--weight_decay\", \"-w\", type=float, default=0.0002, help=\"The weight decay parameter, default=0.0002\")\n parser.add(\"--momentum\", type=float, default=0.9, help=\"The momentum parameter, default=0.9\")\n parser.add(\"--replay_m\", \"-m\", type=int, default=8, help=\"Number of steps to repeat trainig on the same batch, default=8\")\n parser.add(\"--eval_examples\", type=int, default=10000, help=\"Number of evaluation examples, default=10000\")\n parser.add(\"--eval_size\", type=int, default=128, help=\"Evaluation batch size, default=128\")\n parser.add(\"--eval_cpu\", type=bool, default=False, help=\"Set True to do evaluation on CPU instead of GPU, default=False\")\n # params regarding attack\n parser.add(\"--epsilon\", \"-e\", type=float, default=8.0, help=\"Epsilon (Lp Norm distance from the original image) for generating adversarial examples, default=8.0\")\n parser.add(\"--pgd_steps\", \"-k\", type=int, default=20, help=\"Number of steps to PGD attack, default=20\")\n parser.add(\"--step_size\", \"-s\", type=float, default=2.0, help=\"Step size in PGD attack for generating adversarial examples in each step, default=2.0\")\n parser.add(\"--loss_func\", \"-f\", type=str, default=\"xent\", choices=[\"xent\", \"cw\"], help=\"Loss function for the model, choices are [xent, cw], default=xent\")\n parser.add(\"--num_restarts\", type=int, default=1, help=\"Number of resets for the PGD attack, default=1\")\n args = parser.parse_args()\n return args\n\n\nif __name__ == \"__main__\": \n print(get_args())\n pdb.set_trace()\n\n# TODO Default for model_dir\n# TODO Need to update the helps\n" }, { "path": "config.yaml", "content": "" }, { "path": "free_model.py", "content": "# based on https://github.com/tensorflow/models/tree/master/resnet\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport numpy as np\nimport tensorflow as tf\nimport json\n\n\nclass Model(object):\n \"\"\"ResNet model.\"\"\"\n\n def __init__(self, mode, dataset, train_batch_size=None):\n \"\"\"ResNet constructor.\n\n Args:\n mode: One of 'train' and 'eval'.\n \"\"\"\n self.neck = None\n self.y_pred = None\n self.mode = mode\n self.pert = True if mode == 'train' else False\n self.num_classes = 100 if dataset == 'cifar100' else 10\n self.train_batch_size = train_batch_size\n self._build_model()\n\n def add_internal_summaries(self):\n pass\n\n def _stride_arr(self, stride):\n \"\"\"Map a stride scalar to the stride array for tf.nn.conv2d.\"\"\"\n return [1, stride, stride, 1]\n\n def _build_model(self):\n assert self.mode == 'train' or self.mode == 'eval'\n \"\"\"Build the core model within the graph.\"\"\"\n with tf.variable_scope('input'):\n\n self.x_input = tf.placeholder(\n tf.float32,\n shape=[None, 32, 32, 3])\n\n self.y_input = tf.placeholder(tf.int64, shape=None)\n\n if self.pert:\n self.pert = tf.get_variable(name='instance_perturbation', initializer=tf.zeros_initializer,\n shape=[self.train_batch_size, 32, 32, 3], dtype=tf.float32,\n trainable=True)\n self.final_input = self.x_input + self.pert\n self.final_input = tf.clip_by_value(self.final_input, 0., 255.)\n else:\n self.final_input = self.x_input\n\n input_standardized = tf.map_fn(lambda img: tf.image.per_image_standardization(img), self.final_input)\n x = self._conv('init_conv', input_standardized, 3, 3, 16, self._stride_arr(1))\n\n strides = [1, 2, 2]\n activate_before_residual = [True, False, False]\n res_func = self._residual\n\n # Uncomment the following codes to use w28-10 wide residual network.\n # It is more memory efficient than very deep residual network and has\n # comparably good performance.\n # https://arxiv.org/pdf/1605.07146v1.pdf\n filters = [16, 160, 320, 640]\n\n # Update hps.num_residual_units to 9\n\n with tf.variable_scope('unit_1_0'):\n x = res_func(x, filters[0], filters[1], self._stride_arr(strides[0]),\n activate_before_residual[0])\n for i in range(1, 5):\n with tf.variable_scope('unit_1_%d' % i):\n x = res_func(x, filters[1], filters[1], self._stride_arr(1), False)\n\n with tf.variable_scope('unit_2_0'):\n x = res_func(x, filters[1], filters[2], self._stride_arr(strides[1]),\n activate_before_residual[1])\n for i in range(1, 5):\n with tf.variable_scope('unit_2_%d' % i):\n x = res_func(x, filters[2], filters[2], self._stride_arr(1), False)\n\n with tf.variable_scope('unit_3_0'):\n x = res_func(x, filters[2], filters[3], self._stride_arr(strides[2]),\n activate_before_residual[2])\n for i in range(1, 5):\n with tf.variable_scope('unit_3_%d' % i):\n x = res_func(x, filters[3], filters[3], self._stride_arr(1), False)\n\n with tf.variable_scope('unit_last'):\n x = self._batch_norm('final_bn', x)\n x = self._relu(x, 0.1)\n x = self._global_avg_pool(x)\n self.neck = x\n\n with tf.variable_scope('logit'):\n self.pre_softmax = self._fully_connected(x, self.num_classes)\n\n self.predictions = tf.argmax(self.pre_softmax, 1)\n self.y_pred = self.predictions\n self.correct_prediction = tf.equal(self.predictions, self.y_input)\n self.num_correct = tf.reduce_sum(tf.cast(self.correct_prediction, tf.int64))\n self.accuracy = tf.reduce_mean(tf.cast(self.correct_prediction, tf.float32))\n\n with tf.variable_scope('costs'):\n self.y_xent = tf.nn.sparse_softmax_cross_entropy_with_logits(\n logits=self.pre_softmax, labels=self.y_input)\n self.xent = tf.reduce_sum(self.y_xent, name='y_xent')\n self.mean_xent = tf.reduce_mean(self.y_xent)\n self.weight_decay_loss = self._decay()\n\n def _batch_norm(self, name, x):\n \"\"\"Batch normalization.\"\"\"\n with tf.name_scope(name):\n return tf.contrib.layers.batch_norm(inputs=x, decay=.9, center=True, scale=True, activation_fn=None,\n updates_collections=None, is_training=(self.mode == 'train'))\n\n def _residual(self, x, in_filter, out_filter, stride, activate_before_residual=False):\n \"\"\"Residual unit with 2 sub layers.\"\"\"\n if activate_before_residual:\n with tf.variable_scope('shared_activation'):\n x = self._batch_norm('init_bn', x)\n x = self._relu(x, 0.1)\n orig_x = x\n else:\n with tf.variable_scope('residual_only_activation'):\n orig_x = x\n x = self._batch_norm('init_bn', x)\n x = self._relu(x, 0.1)\n\n with tf.variable_scope('sub1'):\n x = self._conv('conv1', x, 3, in_filter, out_filter, stride)\n\n with tf.variable_scope('sub2'):\n x = self._batch_norm('bn2', x)\n x = self._relu(x, 0.1)\n x = self._conv('conv2', x, 3, out_filter, out_filter, [1, 1, 1, 1])\n\n with tf.variable_scope('sub_add'):\n if in_filter != out_filter:\n orig_x = tf.nn.avg_pool(orig_x, stride, stride, 'VALID')\n orig_x = tf.pad(\n orig_x, [[0, 0], [0, 0], [0, 0],\n [(out_filter - in_filter) // 2, (out_filter - in_filter) // 2]])\n x += orig_x\n\n tf.logging.debug('image after unit %s', x.get_shape())\n return x\n\n def _decay(self):\n \"\"\"L2 weight decay loss.\"\"\"\n costs = []\n for var in tf.trainable_variables():\n if var.op.name.find('DW') > 0:\n costs.append(tf.nn.l2_loss(var))\n return tf.add_n(costs)\n\n def _conv(self, name, x, filter_size, in_filters, out_filters, strides):\n \"\"\"Convolution.\"\"\"\n with tf.variable_scope(name):\n n = filter_size * filter_size * out_filters\n kernel = tf.get_variable(\n 'DW', [filter_size, filter_size, in_filters, out_filters],\n tf.float32, initializer=tf.random_normal_initializer(\n stddev=np.sqrt(2.0 / n)))\n return tf.nn.conv2d(x, kernel, strides, padding='SAME')\n\n def _relu(self, x, leakiness=0.0):\n \"\"\"Relu, with optional leaky support.\"\"\"\n return tf.where(tf.less(x, 0.0), leakiness * x, x, name='leaky_relu')\n\n def _fully_connected(self, x, out_dim):\n \"\"\"FullyConnected layer for final output.\"\"\"\n num_non_batch_dimensions = len(x.shape)\n prod_non_batch_dimensions = 1\n for ii in range(num_non_batch_dimensions - 1):\n prod_non_batch_dimensions *= int(x.shape[ii + 1])\n x = tf.reshape(x, [tf.shape(x)[0], -1])\n w = tf.get_variable(\n 'DW', [prod_non_batch_dimensions, out_dim],\n initializer=tf.uniform_unit_scaling_initializer(factor=1.0))\n b = tf.get_variable('biases', [out_dim],\n initializer=tf.constant_initializer())\n return tf.nn.xw_plus_b(x, w, b)\n\n def _global_avg_pool(self, x):\n assert x.get_shape().ndims == 4\n return tf.reduce_mean(x, [1, 2])\n" }, { "path": "free_train.py", "content": "\"\"\"Trains a model, saving checkpoints and tensorboard summaries along\r\n the way.\"\"\"\r\nfrom __future__ import absolute_import\r\nfrom __future__ import division\r\nfrom __future__ import print_function\r\n\r\nfrom datetime import datetime\r\nimport os\r\nimport shutil\r\nfrom timeit import default_timer as timer\r\nimport tensorflow as tf\r\nimport numpy as np\r\nimport sys\r\nfrom free_model import Model\r\nimport cifar10_input\r\nimport cifar100_input\r\nimport pdb\r\n\r\nimport config\r\n\r\ndef get_path_dir(data_dir, dataset, **_):\r\n path = os.path.join(data_dir, dataset)\r\n if os.path.islink(path):\r\n path = os.readlink(path)\r\n return path\r\n\r\n\r\ndef train(tf_seed, np_seed, train_steps, out_steps, summary_steps, checkpoint_steps, step_size_schedule,\r\n weight_decay, momentum, train_batch_size, epsilon, replay_m, model_dir, dataset, **kwargs):\r\n tf.set_random_seed(tf_seed)\r\n np.random.seed(np_seed)\r\n\r\n model_dir = model_dir + '%s_m%d_eps%.1f_b%d' % (dataset, replay_m, epsilon, train_batch_size) # TODO Replace with not defaults\r\n\r\n # Setting up the data and the model\r\n data_path = get_path_dir(dataset=dataset, **kwargs)\r\n if dataset == 'cifar10':\r\n raw_data = cifar10_input.CIFAR10Data(data_path)\r\n else:\r\n raw_data = cifar100_input.CIFAR100Data(data_path)\r\n global_step = tf.contrib.framework.get_or_create_global_step()\r\n model = Model(mode='train', dataset=dataset, train_batch_size=train_batch_size)\r\n\r\n # Setting up the optimizer\r\n boundaries = [int(sss[0]) for sss in step_size_schedule][1:]\r\n values = [sss[1] for sss in step_size_schedule]\r\n learning_rate = tf.train.piecewise_constant(tf.cast(global_step, tf.int32), boundaries, values)\r\n optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)\r\n\r\n # Optimizing computation\r\n total_loss = model.mean_xent + weight_decay * model.weight_decay_loss\r\n grads = optimizer.compute_gradients(total_loss)\r\n\r\n # Compute new image\r\n pert_grad = [g for g, v in grads if 'perturbation' in v.name]\r\n sign_pert_grad = tf.sign(pert_grad[0])\r\n new_pert = model.pert + epsilon * sign_pert_grad\r\n clip_new_pert = tf.clip_by_value(new_pert, -epsilon, epsilon)\r\n assigned = tf.assign(model.pert, clip_new_pert)\r\n\r\n # Train\r\n no_pert_grad = [(tf.zeros_like(v), v) if 'perturbation' in v.name else (g, v) for g, v in grads]\r\n with tf.control_dependencies([assigned]):\r\n min_step = optimizer.apply_gradients(no_pert_grad, global_step=global_step)\r\n tf.initialize_variables([model.pert]) # TODO: Removed from TF\r\n\r\n # Setting up the Tensorboard and checkpoint outputs\r\n if not os.path.exists(model_dir):\r\n os.makedirs(model_dir)\r\n\r\n saver = tf.train.Saver(max_to_keep=1)\r\n tf.summary.scalar('accuracy', model.accuracy)\r\n tf.summary.scalar('xent', model.xent / train_batch_size)\r\n tf.summary.scalar('total loss', total_loss / train_batch_size)\r\n merged_summaries = tf.summary.merge_all()\r\n\r\n gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=1.0)\r\n with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:\r\n print('\\n\\n********** free training for epsilon=%.1f using m_replay=%d **********\\n\\n' % (epsilon, replay_m))\r\n print('important params >>> \\n model dir: %s \\n dataset: %s \\n training batch size: %d \\n' % (model_dir, dataset, train_batch_size))\r\n if dataset == 'cifar100':\r\n print('the ride for CIFAR100 is bumpy -- fasten your seatbelts! \\n \\\r\n you will probably see the training and validation accuracy fluctuating a lot early in trainnig \\n \\\r\n this is natural especially for large replay_m values because we see that mini-batch so many times.')\r\n # initialize data augmentation\r\n if dataset == 'cifar10':\r\n data = cifar10_input.AugmentedCIFAR10Data(raw_data, sess, model)\r\n else:\r\n data = cifar100_input.AugmentedCIFAR100Data(raw_data, sess, model)\r\n\r\n # Initialize the summary writer, global variables, and our time counter.\r\n summary_writer = tf.summary.FileWriter(model_dir + '/train', sess.graph)\r\n eval_summary_writer = tf.summary.FileWriter(model_dir + '/eval')\r\n sess.run(tf.global_variables_initializer())\r\n\r\n # Main training loop\r\n for ii in range(train_steps):\r\n if ii % replay_m == 0:\r\n x_batch, y_batch = data.train_data.get_next_batch(train_batch_size, multiple_passes=True)\r\n nat_dict = {model.x_input: x_batch, model.y_input: y_batch}\r\n\r\n x_eval_batch, y_eval_batch = data.eval_data.get_next_batch(train_batch_size, multiple_passes=True)\r\n eval_dict = {model.x_input: x_eval_batch, model.y_input: y_eval_batch}\r\n\r\n # Output to stdout\r\n if ii % summary_steps == 0:\r\n train_acc, summary = sess.run([model.accuracy, merged_summaries], feed_dict=nat_dict)\r\n summary_writer.add_summary(summary, global_step.eval(sess))\r\n val_acc, summary = sess.run([model.accuracy, merged_summaries], feed_dict=eval_dict)\r\n eval_summary_writer.add_summary(summary, global_step.eval(sess))\r\n print('Step {}: ({})'.format(ii, datetime.now()))\r\n print(' training nat accuracy {:.4}% -- validation nat accuracy {:.4}%'.format(train_acc * 100,\r\n val_acc * 100))\r\n sys.stdout.flush()\r\n # Tensorboard summaries\r\n elif ii % out_steps == 0:\r\n nat_acc = sess.run(model.accuracy, feed_dict=nat_dict)\r\n print('Step {}: ({})'.format(ii, datetime.now()))\r\n print(' training nat accuracy {:.4}%'.format(nat_acc * 100))\r\n\r\n # Write a checkpoint\r\n if (ii+1) % checkpoint_steps == 0:\r\n saver.save(sess, os.path.join(model_dir, 'checkpoint'), global_step=global_step)\r\n\r\n # Actual training step\r\n sess.run(min_step, feed_dict=nat_dict)\r\n\r\n\r\nif __name__ == '__main__':\r\n args = config.get_args()\r\n train(**vars(args))\r\n" }, { "path": "multi_restart_pgd_attack.py", "content": "\"\"\"\r\nImplementation of attack methods. Running this file as a program will\r\nevaluate the model and get the validation accuracy and then\r\napply the attack to the model specified by the config file and store\r\nthe examples in an .npy file.\r\n\"\"\"\r\nfrom __future__ import absolute_import\r\nfrom __future__ import division\r\nfrom __future__ import print_function\r\n\r\nimport tensorflow as tf\r\nimport numpy as np\r\nimport sys\r\nimport cifar10_input\r\nimport cifar100_input\r\nimport config\r\nfrom tqdm import tqdm\r\nimport os\r\n\r\nconfig = config.get_args()\r\n_NUM_RESTARTS = config.num_restarts\r\n\r\n\r\nclass LinfPGDAttack:\r\n def __init__(self, model, epsilon, num_steps, step_size, loss_func):\r\n \"\"\"Attack parameter initialization. The attack performs k steps of\r\n size a, while always staying within epsilon from the initial\r\n point.\"\"\"\r\n self.model = model\r\n self.epsilon = epsilon\r\n self.num_steps = num_steps\r\n self.step_size = step_size\r\n\r\n if loss_func == 'xent':\r\n loss = model.xent\r\n elif loss_func == 'cw':\r\n label_mask = tf.one_hot(model.y_input,\r\n 10,\r\n on_value=1.0,\r\n off_value=0.0,\r\n dtype=tf.float32)\r\n correct_logit = tf.reduce_sum(label_mask * model.pre_softmax, axis=1)\r\n wrong_logit = tf.reduce_max((1 - label_mask) * model.pre_softmax - 1e4 * label_mask, axis=1)\r\n loss = -tf.nn.relu(correct_logit - wrong_logit + 0)\r\n else:\r\n print('Unknown loss function. Defaulting to cross-entropy')\r\n loss = model.xent\r\n\r\n self.grad = tf.gradients(loss, model.x_input)[0]\r\n\r\n def perturb(self, x_nat, y, sess):\r\n \"\"\"Given a set of examples (x_nat, y), returns a set of adversarial\r\n examples within epsilon of x_nat in l_infinity norm.\"\"\"\r\n x = x_nat + np.random.uniform(-self.epsilon, self.epsilon, x_nat.shape)\r\n x = np.clip(x, 0, 255)\r\n\r\n for i in range(self.num_steps):\r\n grad = sess.run(self.grad, feed_dict={self.model.x_input: x,\r\n self.model.y_input: y})\r\n\r\n x = np.add(x, self.step_size * np.sign(grad), out=x, casting='unsafe')\r\n\r\n x = np.clip(x, x_nat - self.epsilon, x_nat + self.epsilon)\r\n x = np.clip(x, 0, 255) # ensure valid pixel range\r\n\r\n return x\r\n\r\ndef get_path_dir(data_dir, dataset, **_):\r\n path = os.path.join(data_dir, dataset)\r\n if os.path.islink(path):\r\n path = os.readlink(path)\r\n return path\r\n\r\n\r\nif __name__ == '__main__':\r\n import sys\r\n import math\r\n from free_model import Model\r\n\r\n model_file = tf.train.latest_checkpoint(config.model_dir)\r\n if model_file is None:\r\n print('No model found')\r\n sys.exit()\r\n\r\n dataset = config.dataset\r\n data_dir = config.data_dir\r\n data_path = get_path_dir(data_dir, dataset)\r\n\r\n model = Model(mode='eval', dataset=dataset)\r\n attack = LinfPGDAttack(model,\r\n config.epsilon,\r\n config.pgd_steps,\r\n config.step_size,\r\n config.loss_func)\r\n saver = tf.train.Saver()\r\n\r\n\r\n if dataset == 'cifar10': \r\n cifar = cifar10_input.CIFAR10Data(data_path)\r\n else:\r\n cifar = cifar100_input.CIFAR100Data(data_path)\r\n\r\n with tf.Session() as sess:\r\n # Restore the checkpoint\r\n saver.restore(sess, model_file)\r\n\r\n # Iterate over the samples batch-by-batch\r\n num_eval_examples = config.eval_examples\r\n eval_batch_size = config.eval_size\r\n num_batches = int(math.ceil(num_eval_examples / eval_batch_size))\r\n\r\n x_adv = [] # adv accumulator\r\n\r\n print('getting clean validation accuracy')\r\n total_corr = 0\r\n for ibatch in tqdm(range(num_batches)):\r\n bstart = ibatch * eval_batch_size\r\n bend = min(bstart + eval_batch_size, num_eval_examples)\r\n\r\n x_batch = cifar.eval_data.xs[bstart:bend, :].astype(np.float32)\r\n y_batch = cifar.eval_data.ys[bstart:bend]\r\n\r\n dict_val = {model.x_input: x_batch, model.y_input: y_batch}\r\n cur_corr = sess.run(model.num_correct, feed_dict=dict_val)\r\n total_corr += cur_corr\r\n print('** validation accuracy: %.3f **\\n\\n' % (total_corr / float(num_eval_examples) * 100))\r\n\r\n print('Iterating over {} batches'.format(num_batches))\r\n\r\n total_corr, total_num = 0, 0\r\n for ibatch in range(num_batches):\r\n bstart = ibatch * eval_batch_size\r\n bend = min(bstart + eval_batch_size, num_eval_examples)\r\n curr_num = bend - bstart\r\n total_num += curr_num\r\n print('mini batch: {}/{} -- batch size: {}'.format(ibatch + 1, num_batches, curr_num))\r\n sys.stdout.flush()\r\n\r\n x_batch = cifar.eval_data.xs[bstart:bend, :].astype(np.float32)\r\n y_batch = cifar.eval_data.ys[bstart:bend]\r\n\r\n best_batch_adv = np.copy(x_batch)\r\n dict_adv = {model.x_input: best_batch_adv, model.y_input: y_batch}\r\n cur_corr, y_pred_batch, best_loss = sess.run([model.num_correct, model.predictions, model.y_xent],\r\n feed_dict=dict_adv)\r\n for ri in range(_NUM_RESTARTS):\r\n x_batch_adv = attack.perturb(x_batch, y_batch, sess)\r\n dict_adv = {model.x_input: x_batch_adv, model.y_input: y_batch}\r\n cur_corr, y_pred_batch, this_loss = sess.run([model.num_correct, model.predictions, model.y_xent],\r\n feed_dict=dict_adv)\r\n bb = best_loss >= this_loss\r\n bw = best_loss < this_loss\r\n best_batch_adv[bw, :, :, :] = x_batch_adv[bw, :, :, :]\r\n\r\n best_corr, y_pred_batch, best_loss = sess.run([model.num_correct, model.predictions, model.y_xent],\r\n feed_dict={model.x_input: best_batch_adv,\r\n model.y_input: y_batch})\r\n print('restart %d: num correct: %d -- loss:%.4f' % (ri, best_corr, np.mean(best_loss)))\r\n total_corr += best_corr\r\n print('accuracy till now {:4}% \\n\\n'.format(float(total_corr) / total_num * 100))\r\n\r\n x_adv.append(best_batch_adv)\r\n\r\n x_adv = np.concatenate(x_adv, axis=0)\r\n" }, { "path": "requirements/base.txt", "content": "ConfigArgParse==0.14.0\ntqdm==4.31.1\n" }, { "path": "requirements/cpu.txt", "content": "-r base.txt\ntensorflow>=0.10.0\n" }, { "path": "requirements/gpu.txt", "content": "-r base.txt\ntensorflow-gpu>=0.10.0\n" } ]