Repository: huanzhang12/ZOO-Attack Branch: master Commit: 31c41cda71e9 Files: 24 Total size: 321.5 KB Directory structure: gitextract_bj_2e_i4/ ├── COPYRIGHT ├── LICENSE ├── README.md ├── cifar_blackbox.py ├── l0_attack.py ├── l2_attack.py ├── l2_attack_black.py ├── labels/ │ ├── imagenet_val_to_carlini.py │ ├── label2num.txt │ ├── labels.py │ ├── labels.txt │ └── synset_words.txt ├── li_attack.py ├── mnist_blackbox.py ├── retrain.py ├── setup_cifar.py ├── setup_inception.py ├── setup_mnist.py ├── substitute_blackbox.py ├── test_all.py ├── test_attack.py ├── test_attack_black.py ├── train_models.py └── verify.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: COPYRIGHT ================================================ Copyright (C) 2017-2018, IBM Corp. Copyright (c) 2016 Nicholas Carlini Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ **As requested by IBM, this repository is moved to https://github.com/IBM/ZOO-Attack, but we aim to keep both repositories synced up.** The code is released under Apache License v2. ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks ===================================== ZOO is a **z**eroth **o**rder **o**ptimization based attack to attack deep neural networks (DNNs). We propose an effective black-box attack that only requires access to the input (images) and the output (confidence scores) of a targeted DNN. We formularize the attack as an optimization problem (similar as Carlini and Wagner's attack), and propose a new loss function suitable for the black-box setting. We use zeroth order stochastic coordinate descent to optimize on the target DNN directly, along with dimension reduction, hierarchical attack and importance sampling techniques to make the attack efficient. No transferability or substitute model is required. There are two variants of ZOO, ZOO-ADAM and ZOO-Newton, corresponding to different solvers (ADAM and Newton) to find the best coordinate update. In practice ZOO-ADAM usually works better with fine-tuned parameters, but ZOO-Newton is more stable when close to the optimal solution. The experiment code is based on Carlini and Wagner's L2 attack, with zeroth order optimizer added in `l2_attack_black.py`. The inception model is updated to a new version (`inception_v3_2016_08_28.tar.gz`), and an unified interface `test_all.py` is added. For more details, please see our paper: [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://arxiv.org/abs/1708.03999) by Pin-Yu Chen\*, Huan Zhang\*, Yash Sharma, Jinfeng Yi, Cho-Jui Hsieh \* Equal contribution Setup and train models ------------------------------------- The code is tested with python3 and TensorFlow v1.2 and v1.3. The following packages are required: ``` sudo apt-get install python3-pip sudo pip3 install --upgrade pip sudo pip3 install pillow scipy numpy tensorflow-gpu keras h5py numba ``` Prepare the MNIST and CIFAR-10 data and models for attack: ``` python3 train_models.py ``` To download the inception model: ``` python3 setup_inception.py ``` To prepare the ImageNet dataset, download and unzip the following archive: http://download.huan-zhang.com/datasets/adv/img.tar.gz and put the `imgs` folder in `../imagenetdata`. This path can be changed in `setup_inception.py`. Run attacks -------------------------------------- An unified attack interface, `test_all.py` is provided. Run `python3 test_all.py -h` to get a list of arguments and help. The following are some examples of attacks: Run ZOO black-box targeted attack, on the mnist dataset with 200 images, with ZOO-ADAM solver, search for best regularization constant for 9 iterations, and save attack images to folder `black_results`. To run on the CIFAR-10 dataset, replace 'mnist' with 'cifar10'. ``` python3 test_all.py -a black -d mnist -n 200 --solver adam -b 9 -s "black_results" ``` Run Carlini and Wagner's white-box targeted attack, on the mnist dataset with 200 images, using the Z (logits) value in objective (only available in white-box setting), search for best regularization constant for 9 iterations, and save attack images to folder `white_results`. ``` python3 test_all.py -a white -d mnist -n 200 --use_zvalue -b 9 -s "white_results" ``` Run ZOO black-box *untargeted* attack, on the imagenet dataset with 150 images, with ZOO-ADAM solver, do not binary search the regularization parameter (i.e., search only 1 time), and set the initial regularization parameter to a fixed value (10.0). Use attack-space dimension reduction with image resizing, and reset ADAM states when the first attack is found. Run a maximum of 1500 iterations, and print out loss every 10 iterations. Save attack images to folder `imagenet_untargeted`. ``` python3 test_all.py --untargeted -a black -d imagenet -n 150 --solver adam -b 1 -c 10.0 --use_resize --reset_adam -m 1500 -p 10 -s "imagenet_untargeted" ``` Run ZOO black-box targeted attack, on the imagenet dataset, with the 69th image only. Set the regularization parameter to 10.0 and do not binary search. Use attack-space dimension reduction and hierarchical attack with image resizing, and reset ADAM states when the first attack is found. Run a maximum of 20000 iterations, and print out loss every 10 iterations. Save attack images to folder `imagenet_all_tricks_img69`. ``` python3 test_all.py -a black --solver adam -d imagenet -f 69 -n 1 -c 10.0 --use_resize --reset_adam -m 20000 -p 10 -s "imagenet_all_tricks_img69" ``` Importance sampling is on by default for ImageNet data, and can be turned off by `--uniform` option. To change the hierarchical attack dimension scheduling, change `l2_attack_black.py`, near line 580. ================================================ FILE: cifar_blackbox.py ================================================ ## Copyright (C) IBM Corp, 2017-2018 from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals import numpy as np from six.moves import xrange import keras from keras import backend from keras.utils.np_utils import to_categorical from keras.models import Sequential from keras.layers import Dense, Flatten, Activation, Dropout from keras.datasets import cifar10 from keras.utils import np_utils import tensorflow as tf from tensorflow.python.platform import app from tensorflow.python.platform import flags from cleverhans.utils_keras import cnn_model from cleverhans.utils_tf import model_train, model_eval, batch_eval, tf_model_load from cleverhans.attacks import FastGradientMethod from cleverhans.attacks_tf import jacobian_graph, jacobian_augmentation from cleverhans.utils_keras import KerasModelWrapper from setup_cifar import CIFARModel FLAGS = flags.FLAGS def data_cifar10(): """ Preprocess CIFAR10 dataset :return: """ # These values are specific to CIFAR10 img_rows = 32 img_cols = 32 nb_classes = 10 # the data, shuffled and split between train and test sets (X_train, y_train), (X_test, y_test) = cifar10.load_data() if keras.backend.image_dim_ordering() == 'th': X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols) X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols) else: X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3) X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3) X_train = X_train.astype('float32') X_test = X_test.astype('float32') X_train /= 255 X_test /= 255 print('X_train shape:', X_train.shape) print(X_train.shape[0], 'train samples') print(X_test.shape[0], 'test samples') # convert class vectors to binary class matrices Y_train = np_utils.to_categorical(y_train, nb_classes) Y_test = np_utils.to_categorical(y_test, nb_classes) return X_train, Y_train, X_test, Y_test def setup_tutorial(): """ Helper function to check correct configuration of tf and keras for tutorial :return: True if setup checks completed """ # Set TF random seed to improve reproducibility tf.set_random_seed(1234) if not hasattr(backend, "tf"): raise RuntimeError("This tutorial requires keras to be configured" " to use the TensorFlow backend.") # Image dimensions ordering should follow the Theano convention if keras.backend.image_dim_ordering() != 'tf': keras.backend.set_image_dim_ordering('tf') print("INFO: '~/.keras/keras.json' sets 'image_dim_ordering' " "to 'th', temporarily setting to 'tf'") return True def prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate): """ Define and train a model that simulates the "remote" black-box oracle described in the original paper. :param sess: the TF session :param x: the input placeholder for CIFAR :param y: the ouput placeholder for CIFAR :param X_train: the training data for the oracle :param Y_train: the training labels for the oracle :param X_test: the testing data for the oracle :param Y_test: the testing labels for the oracle :param nb_epochs: number of epochs to train model :param batch_size: size of training batches :param learning_rate: learning rate for training :return: """ # Define TF model graph (for the black-box model) model = CIFARModel(use_log = True).model # model = CIFARModel(use_log = True).model predictions = model(x) print("Defined TensorFlow model graph.") # Train an CIFAR model if FLAGS.load_pretrain: # use the restored CIFAR model tf_model_load(sess) else: train_params = { 'nb_epochs': nb_epochs, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, predictions, X_train, Y_train, verbose=True, save=True, args=train_params) # Print out the accuracy on legitimate data eval_params = {'batch_size': batch_size} accuracy = model_eval(sess, x, y, predictions, X_test, Y_test, args=eval_params) print('Test accuracy of black-box on legitimate test ' 'examples: ' + str(accuracy)) return model, predictions, accuracy def train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda): """ This function creates the substitute by alternatively augmenting the training data and training the substitute. :param sess: TF session :param x: input TF placeholder :param y: output TF placeholder :param bbox_preds: output of black-box model predictions :param X_sub: initial substitute training data :param Y_sub: initial substitute training labels :param nb_classes: number of output classes :param nb_epochs_s: number of epochs to train substitute model :param batch_size: size of training batches :param learning_rate: learning rate for training :param data_aug: number of times substitute training data is augmented :param lmbda: lambda from arxiv.org/abs/1602.02697 :return: """ # Define TF model graph (for the black-box model) model_sub = CIFARModel(use_log = True).model preds_sub = model_sub(x) print("Defined TensorFlow model graph for the substitute.") # Define the Jacobian symbolically using TensorFlow grads = jacobian_graph(preds_sub, x, nb_classes) # Train the substitute and augment dataset alternatively for rho in xrange(data_aug): print("Substitute training epoch #" + str(rho)) train_params = { 'nb_epochs': nb_epochs_s, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, preds_sub, X_sub, to_categorical(Y_sub), init_all=False, verbose=False, args=train_params) # If we are not at last substitute training iteration, augment dataset if rho < data_aug - 1: print("Augmenting substitute training data.") # Perform the Jacobian augmentation X_sub = jacobian_augmentation(sess, x, X_sub, Y_sub, grads, lmbda) print("Labeling substitute training data.") # Label the newly generated synthetic points using the black-box Y_sub = np.hstack([Y_sub, Y_sub]) X_sub_prev = X_sub[int(len(X_sub)/2):] eval_params = {'batch_size': batch_size} bbox_val = batch_eval(sess, [x], [bbox_preds], [X_sub_prev], args=eval_params)[0] # Note here that we take the argmax because the adversary # only has access to the label (not the probabilities) output # by the black-box model Y_sub[int(len(X_sub)/2):] = np.argmax(bbox_val, axis=1) return model_sub, preds_sub def cifar_blackbox(train_start=0, train_end=60000, test_start=0, test_end=10000, nb_classes=10, batch_size=128, learning_rate=0.001, nb_epochs=50, holdout=150, data_aug=6, nb_epochs_s=50, lmbda=0.1): """ CIFAR tutorial for the black-box attack from arxiv.org/abs/1602.02697 :param train_start: index of first training set example :param train_end: index of last training set example :param test_start: index of first test set example :param test_end: index of last test set example :return: a dictionary with: * black-box model accuracy on test set * substitute model accuracy on test set * black-box model accuracy on adversarial examples transferred from the substitute model """ keras.layers.core.K.set_learning_phase(0) # Dictionary used to keep track and return key accuracies accuracies = {} # Perform tutorial setup assert setup_tutorial() # Create TF session and set as Keras backend session gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.3) sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) keras.backend.set_session(sess) # Get CIFAR data X_train, Y_train, X_test, Y_test = data_cifar10() # Initialize substitute training set reserved for adversary X_sub = X_test[:holdout] Y_sub = np.argmax(Y_test[:holdout], axis=1) # Redefine test set as remaining samples unavailable to adversaries X_test = X_test[holdout:] Y_test = Y_test[holdout:] # Define input and output TF placeholders x = tf.placeholder(tf.float32, shape=(None, 32, 32, 3)) y = tf.placeholder(tf.float32, shape=(None, 10)) # Simulate the black-box model locally # You could replace this by a remote labeling API for instance print("Preparing the black-box model.") prep_bbox_out = prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate) model, bbox_preds, accuracies['bbox'] = prep_bbox_out # Train substitute using method from https://arxiv.org/abs/1602.02697 print("Training the substitute model.") train_sub_out = train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda) model_sub, preds_sub = train_sub_out # Evaluate the substitute model on clean test examples eval_params = {'batch_size': batch_size} acc = model_eval(sess, x, y, preds_sub, X_test, Y_test, args=eval_params) accuracies['sub'] = acc print('substitution model accuracy:', acc) # Find the correctly predicted labels original_predict = batch_eval(sess, [x], [bbox_preds], [X_test], args=eval_params)[0] original_class = np.argmax(original_predict, axis = 1) true_class = np.argmax(Y_test, axis = 1) mask = true_class == original_class print(np.sum(mask), "out of", mask.size, "are correct labeled,", len(X_test[mask])) # Initialize the Fast Gradient Sign Method (FGSM) attack object. fgsm_par = {'eps': 0.4, 'ord': np.inf, 'clip_min': 0., 'clip_max': 1.} wrap = KerasModelWrapper(model_sub) fgsm = FastGradientMethod(wrap, sess=sess) # Craft adversarial examples using the substitute eval_params = {'batch_size': batch_size} x_adv_sub = fgsm.generate(x, **fgsm_par) # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, model(x_adv_sub), X_test, Y_test, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute: ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex'] = accuracy # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, bbox_preds, X_test[mask], Y_test[mask], args=eval_params) print('Test accuracy of excluding originally incorrect labels: ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc_ori'] = accuracy # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, model(x_adv_sub), X_test[mask], Y_test[mask], args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (excluding originally incorrect labels): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc'] = accuracy return accuracies def main(argv=None): print(cifar_blackbox(nb_classes=FLAGS.nb_classes, batch_size=FLAGS.batch_size, learning_rate=FLAGS.learning_rate, nb_epochs=FLAGS.nb_epochs, holdout=FLAGS.holdout, data_aug=FLAGS.data_aug, nb_epochs_s=FLAGS.nb_epochs_s, lmbda=FLAGS.lmbda)) if __name__ == '__main__': # General flags flags.DEFINE_integer('nb_classes', 10, 'Number of classes in problem') flags.DEFINE_integer('batch_size', 128, 'Size of training batches') flags.DEFINE_float('learning_rate', 0.0005, 'Learning rate for training') # Flags related to oracle flags.DEFINE_integer('nb_epochs', 50, 'Number of epochs to train model') # Flags related to substitute flags.DEFINE_integer('holdout', 150, 'Test set holdout for adversary') flags.DEFINE_integer('data_aug', 6, 'Nb of substitute data augmentations') flags.DEFINE_integer('nb_epochs_s', 50, 'Training epochs for substitute') flags.DEFINE_float('lmbda', 0.1, 'Lambda from arxiv.org/abs/1602.02697') # Flags related to saving/loading flags.DEFINE_bool('load_pretrain', False, 'load pretrained model from sub_saved/cifar-model') flags.DEFINE_string('train_dir', 'sub_saved', 'model saving path') flags.DEFINE_string('filename', 'cifar-model', 'cifar model name') app.run() ================================================ FILE: l0_attack.py ================================================ ## l0_attack.py -- attack a network optimizing for l_0 distance ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import sys import tensorflow as tf import numpy as np MAX_ITERATIONS = 1000 # number of iterations to perform gradient descent ABORT_EARLY = True # abort gradient descent upon first valid solution LEARNING_RATE = 1e-2 # larger values converge faster to less accurate results INITIAL_CONST = 1e-3 # the first value of c to start at LARGEST_CONST = 2e6 # the largest value of c to go up to before giving up REDUCE_CONST = False # try to lower c each iteration; faster to set to false TARGETED = True # should we target one specific class? or just be wrong? CONST_FACTOR = 2.0 # f>1, rate at which we increase constant, smaller better class CarliniL0: def __init__(self, sess, model, targeted = TARGETED, learning_rate = LEARNING_RATE, max_iterations = MAX_ITERATIONS, abort_early = ABORT_EARLY, initial_const = INITIAL_CONST, largest_const = LARGEST_CONST, reduce_const = REDUCE_CONST, const_factor = CONST_FACTOR, independent_channels = False): """ The L_0 optimized attack. Returns adversarial examples for the supplied model. targeted: True if we should perform a targetted attack, False otherwise. learning_rate: The learning rate for the attack algorithm. Smaller values produce better results but are slower to converge. max_iterations: The maximum number of iterations. Larger values are more accurate; setting too small will require a large learning rate and will produce poor results. abort_early: If true, allows early aborts if gradient descent gets stuck. initial_const: The initial tradeoff-constant to use to tune the relative importance of distance and confidence. Should be set to a very small value (but positive). largest_const: The largest constant to use until we report failure. Should be set to a very large value. const_factor: The rate at which we should increase the constant, when the previous constant failed. Should be greater than one, smaller is better. independent_channels: set to false optimizes for number of pixels changed, set to true (not recommended) returns number of channels changed. """ self.model = model self.sess = sess self.TARGETED = targeted self.LEARNING_RATE = learning_rate self.MAX_ITERATIONS = max_iterations self.ABORT_EARLY = abort_early self.INITIAL_CONST = initial_const self.LARGEST_CONST = largest_const self.REDUCE_CONST = reduce_const self.const_factor = const_factor self.independent_channels = independent_channels self.grad = self.gradient_descent(sess, model) def gradient_descent(self, sess, model): def compare(x,y): if self.TARGETED: return x == y else: return x != y shape = (1,model.image_size,model.image_size,model.num_channels) # the variable to optimize over modifier = tf.Variable(np.zeros(shape,dtype=np.float32)) # the variables we're going to hold, use for efficiency canchange = tf.Variable(np.zeros(shape),dtype=np.float32) simg = tf.Variable(np.zeros(shape,dtype=np.float32)) original = tf.Variable(np.zeros(shape,dtype=np.float32)) timg = tf.Variable(np.zeros(shape,dtype=np.float32)) tlab = tf.Variable(np.zeros((1,model.num_labels),dtype=np.float32)) const = tf.placeholder(tf.float32, []) # and the assignment to set the variables assign_modifier = tf.placeholder(np.float32,shape) assign_canchange = tf.placeholder(np.float32,shape) assign_simg = tf.placeholder(np.float32,shape) assign_original = tf.placeholder(np.float32,shape) assign_timg = tf.placeholder(np.float32,shape) assign_tlab = tf.placeholder(np.float32,(1,self.model.num_labels)) # these are the variables to initialize when we run set_modifier = tf.assign(modifier, assign_modifier) setup = [] setup.append(tf.assign(canchange, assign_canchange)) setup.append(tf.assign(timg, assign_timg)) setup.append(tf.assign(original, assign_original)) setup.append(tf.assign(simg, assign_simg)) setup.append(tf.assign(tlab, assign_tlab)) newimg = (tf.tanh(modifier + simg)/2)*canchange+(1-canchange)*original output = model.predict(newimg) real = tf.reduce_sum((tlab)*output,1) other = tf.reduce_max((1-tlab)*output - (tlab*10000),1) if self.TARGETED: # if targetted, optimize for making the other class most likely loss1 = tf.maximum(0.0, other-real+.01) else: # if untargeted, optimize for making this class least likely. loss1 = tf.maximum(0.0, real-other+.01) # sum up the losses loss2 = tf.reduce_sum(tf.square(newimg-tf.tanh(timg)/2)) loss = const*loss1+loss2 outgrad = tf.gradients(loss, [modifier])[0] # setup the adam optimizer and keep track of variables we're creating start_vars = set(x.name for x in tf.global_variables()) optimizer = tf.train.AdamOptimizer(self.LEARNING_RATE) train = optimizer.minimize(loss, var_list=[modifier]) end_vars = tf.global_variables() new_vars = [x for x in end_vars if x.name not in start_vars] init = tf.variables_initializer(var_list=[modifier,canchange,simg, original,timg,tlab]+new_vars) def doit(oimgs, labs, starts, valid, CONST): # convert to tanh-space imgs = np.arctanh(np.array(oimgs)*1.999999) starts = np.arctanh(np.array(starts)*1.999999) # initialize the variables sess.run(init) sess.run(setup, {assign_timg: imgs, assign_tlab:labs, assign_simg: starts, assign_original: oimgs, assign_canchange: valid}) while CONST < self.LARGEST_CONST: # try solving for each value of the constant print('try const', CONST) for step in range(self.MAX_ITERATIONS): feed_dict={const: CONST} # remember the old value oldmodifier = self.sess.run(modifier) if step%(self.MAX_ITERATIONS//10) == 0: print(step,*sess.run((loss1,loss2),feed_dict=feed_dict)) # perform the update step _, works = sess.run([train, loss1], feed_dict=feed_dict) if works < .0001 and (self.ABORT_EARLY or step == CONST-1): # it worked previously, restore the old value and finish self.sess.run(set_modifier, {assign_modifier: oldmodifier}) grads, scores, nimg = sess.run((outgrad, output,newimg), feed_dict=feed_dict) l2s=np.square(nimg-np.tanh(imgs)/2).sum(axis=(1,2,3)) return grads, scores, nimg, CONST # we didn't succeed, increase constant and try again CONST *= self.const_factor return doit def attack(self, imgs, targets): """ Perform the L_0 attack on the given images for the given targets. If self.targeted is true, then the targets represents the target labels. If self.targeted is false, then targets are the original class labels. """ r = [] for i,(img,target) in enumerate(zip(imgs, targets)): print("Attack iteration",i) r.extend(self.attack_single(img, target)) return np.array(r) def attack_single(self, img, target): """ Run the attack on a single image and label """ # the pixels we can change valid = np.ones((1,self.model.image_size,self.model.image_size,self.model.num_channels)) # the previous image prev = np.copy(img).reshape((1,self.model.image_size,self.model.image_size, self.model.num_channels)) last_solution = None const = self.INITIAL_CONST while True: # try to solve given this valid map res = self.grad([np.copy(img)], [target], np.copy(prev), valid, const) if res == None: # the attack failed, we return this as our final answer print("Final answer",equal_count) return last_solution # the attack succeeded, now we pick new pixels to set to 0 restarted = False gradientnorm, scores, nimg, const = res if self.REDUCE_CONST: const /= 2 equal_count = self.model.image_size**2-np.sum(np.all(np.abs(img-nimg[0])<.0001,axis=2)) print("Forced equal:",np.sum(1-valid), "Equal count:",equal_count) if np.sum(valid) == 0: # if no pixels changed, return return [img] if self.independent_channels: # we are allowed to change each channel independently valid = valid.flatten() totalchange = abs(nimg[0]-img)*np.abs(gradientnorm[0]) else: # we care only about which pixels change, not channels independently # compute total change as sum of change for each channel valid = valid.reshape((self.model.image_size**2,self.model.num_channels)) totalchange = abs(np.sum(nimg[0]-img,axis=2))*np.sum(np.abs(gradientnorm[0]),axis=2) totalchange = totalchange.flatten() # set some of the pixels to 0 depending on their total change did = 0 for e in np.argsort(totalchange): if np.all(valid[e]): did += 1 valid[e] = 0 if totalchange[e] > .01: # if this pixel changed a lot, skip break if did >= .3*equal_count**.5: # if we changed too many pixels, skip break valid = np.reshape(valid,(1,self.model.image_size,self.model.image_size,-1)) print("Now forced equal:",np.sum(1-valid)) last_solution = prev = nimg ================================================ FILE: l2_attack.py ================================================ ## l2_attack.py -- attack a network optimizing for l_2 distance ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import sys import tensorflow as tf import numpy as np import time BINARY_SEARCH_STEPS = 1 # number of times to adjust the constant with binary search MAX_ITERATIONS = 10000 # number of iterations to perform gradient descent ABORT_EARLY = True # if we stop improving, abort gradient descent early LEARNING_RATE = 2e-3 # larger values converge faster to less accurate results TARGETED = True # should we target one specific class? or just be wrong? CONFIDENCE = 0 # how strong the adversarial example should be INITIAL_CONST = 0.01 # the initial constant c to pick as a first guess class CarliniL2: def __init__(self, sess, model, batch_size=1, confidence = CONFIDENCE, targeted = TARGETED, learning_rate = LEARNING_RATE, binary_search_steps = BINARY_SEARCH_STEPS, max_iterations = MAX_ITERATIONS, print_every = 100, early_stop_iters = 0, abort_early = ABORT_EARLY, initial_const = INITIAL_CONST, use_log = False, adam_beta1 = 0.9, adam_beta2 = 0.999): """ The L_2 optimized attack. This attack is the most efficient and should be used as the primary attack to evaluate potential defenses. Returns adversarial examples for the supplied model. confidence: Confidence of adversarial examples: higher produces examples that are farther away, but more strongly classified as adversarial. batch_size: Number of attacks to run simultaneously. targeted: True if we should perform a targetted attack, False otherwise. learning_rate: The learning rate for the attack algorithm. Smaller values produce better results but are slower to converge. binary_search_steps: The number of times we perform binary search to find the optimal tradeoff-constant between distance and confidence. max_iterations: The maximum number of iterations. Larger values are more accurate; setting too small will require a large learning rate and will produce poor results. abort_early: If true, allows early aborts if gradient descent gets stuck. initial_const: The initial tradeoff-constant to use to tune the relative importance of distance and confidence. If binary_search_steps is large, the initial constant is not important. """ image_size, num_channels, num_labels = model.image_size, model.num_channels, model.num_labels self.sess = sess self.TARGETED = targeted self.LEARNING_RATE = learning_rate self.MAX_ITERATIONS = max_iterations self.print_every = print_every self.early_stop_iters = early_stop_iters if early_stop_iters != 0 else max_iterations // 10 print("early stop:", self.early_stop_iters) self.BINARY_SEARCH_STEPS = binary_search_steps self.ABORT_EARLY = abort_early self.CONFIDENCE = confidence self.initial_const = initial_const self.batch_size = batch_size self.repeat = binary_search_steps >= 10 shape = (batch_size,image_size,image_size,num_channels) # the variable we're going to optimize over self.modifier = tf.Variable(np.zeros(shape,dtype=np.float32)) # self.modifier = tf.Variable(np.load('black_iter_350.npy').astype(np.float32).reshape(shape)) # these are variables to be more efficient in sending data to tf self.timg = tf.Variable(np.zeros(shape), dtype=tf.float32) self.tlab = tf.Variable(np.zeros((batch_size,num_labels)), dtype=tf.float32) self.const = tf.Variable(np.zeros(batch_size), dtype=tf.float32) # and here's what we use to assign them self.assign_timg = tf.placeholder(tf.float32, shape) self.assign_tlab = tf.placeholder(tf.float32, (batch_size,num_labels)) self.assign_const = tf.placeholder(tf.float32, [batch_size]) # the resulting image, tanh'd to keep bounded from -0.5 to 0.5 self.newimg = tf.tanh(self.modifier + self.timg)/2 # prediction BEFORE-SOFTMAX of the model self.output = model.predict(self.newimg) # distance to the input data self.l2dist = tf.reduce_sum(tf.square(self.newimg-tf.tanh(self.timg)/2),[1,2,3]) # compute the probability of the label class versus the maximum other self.real = tf.reduce_sum((self.tlab)*self.output,1) self.other = tf.reduce_max((1-self.tlab)*self.output - (self.tlab*10000),1) if self.TARGETED: if use_log: # loss1 = tf.maximum(- tf.log(self.other), - tf.log(self.real)) # loss1 = - tf.log(self.real) loss1 = tf.maximum(0.0, tf.log(self.other + 1e-30) - tf.log(self.real + 1e-30)) else: # if targetted, optimize for making the other class most likely loss1 = tf.maximum(0.0, self.other-self.real+self.CONFIDENCE) else: if use_log: # loss1 = tf.log(self.real) loss1 = tf.maximum(0.0, tf.log(self.real + 1e-30) - tf.log(self.other + 1e-30)) else: # if untargeted, optimize for making this class least likely. loss1 = tf.maximum(0.0, self.real-self.other+self.CONFIDENCE) # sum up the losses self.loss2 = tf.reduce_sum(self.l2dist) self.loss1 = tf.reduce_sum(self.const*loss1) self.loss = self.loss1+self.loss2 # Setup the adam optimizer and keep track of variables we're creating start_vars = set(x.name for x in tf.global_variables()) # optimizer = tf.train.GradientDescentOptimizer(self.LEARNING_RATE) # optimizer = tf.train.MomentumOptimizer(self.LEARNING_RATE, 0.99) # optimizer = tf.train.RMSPropOptimizer(self.LEARNING_RATE) # optimizer = tf.train.AdadeltaOptimizer(self.LEARNING_RATE) optimizer = tf.train.AdamOptimizer(self.LEARNING_RATE, adam_beta1, adam_beta2) self.train = optimizer.minimize(self.loss, var_list=[self.modifier]) end_vars = tf.global_variables() new_vars = [x for x in end_vars if x.name not in start_vars] # these are the variables to initialize when we run self.setup = [] self.setup.append(self.timg.assign(self.assign_timg)) self.setup.append(self.tlab.assign(self.assign_tlab)) self.setup.append(self.const.assign(self.assign_const)) # self.grad_op = tf.gradients(self.loss, self.modifier) self.init = tf.variables_initializer(var_list=[self.modifier]+new_vars) def attack(self, imgs, targets): """ Perform the L_2 attack on the given images for the given targets. If self.targeted is true, then the targets represents the target labels. If self.targeted is false, then targets are the original class labels. """ r = [] print('go up to',len(imgs)) for i in range(0,len(imgs),self.batch_size): print('tick',i) r.extend(self.attack_batch(imgs[i:i+self.batch_size], targets[i:i+self.batch_size])[0]) return np.array(r) def attack_batch(self, imgs, labs): """ Run the attack on a batch of images and labels. """ def compare(x,y): if not isinstance(x, (float, int, np.int64)): x = np.copy(x) if self.TARGETED: x[y] -= self.CONFIDENCE else: x[y] += self.CONFIDENCE x = np.argmax(x) if self.TARGETED: return x == y else: return x != y batch_size = self.batch_size # convert to tanh-space imgs = np.arctanh(imgs*1.999999) # set the lower and upper bounds accordingly lower_bound = np.zeros(batch_size) CONST = np.ones(batch_size)*self.initial_const upper_bound = np.ones(batch_size)*1e10 # the best l2, score, and image attack o_bestl2 = [1e10]*batch_size o_bestscore = [-1]*batch_size o_bestattack = [np.zeros(imgs[0].shape)]*batch_size o_best_const = [self.initial_const]*batch_size for outer_step in range(self.BINARY_SEARCH_STEPS): print("current best l2", o_bestl2) # completely reset adam's internal state. self.sess.run(self.init) batch = imgs[:batch_size] batchlab = labs[:batch_size] bestl2 = [1e10]*batch_size bestscore = [-1]*batch_size # The last iteration (if we run many steps) repeat the search once. if self.repeat == True and outer_step == self.BINARY_SEARCH_STEPS-1: CONST = upper_bound # set the variables so that we don't have to send them over again self.sess.run(self.setup, {self.assign_timg: batch, self.assign_tlab: batchlab, self.assign_const: CONST}) prev = 1e6 train_timer = 0.0 for iteration in range(self.MAX_ITERATIONS): # print out the losses every 10% if iteration%(self.MAX_ITERATIONS//self.print_every) == 0: # print(iteration,self.sess.run((self.loss,self.real,self.other,self.loss1,self.loss2))) # grad = self.sess.run(self.grad_op) # old_modifier = self.sess.run(self.modifier) # np.save('white_iter_{}'.format(iteration), modifier) loss, real, other, loss1, loss2 = self.sess.run((self.loss,self.real,self.other,self.loss1,self.loss2)) print("[STATS][L2] iter = {}, time = {:.3f}, loss = {:.5g}, real = {:.5g}, other = {:.5g}, loss1 = {:.5g}, loss2 = {:.5g}".format(iteration, train_timer, loss, real[0], other[0], loss1, loss2)) sys.stdout.flush() attack_begin_time = time.time() # perform the attack _, l, l2s, scores, nimg = self.sess.run([self.train, self.loss, self.l2dist, self.output, self.newimg]) new_modifier = self.sess.run(self.modifier) # print(grad[0].reshape(-1)) # print((old_modifier - new_modifier).reshape(-1)) # check if we should abort search if we're getting nowhere. if self.ABORT_EARLY and iteration % self.early_stop_iters == 0: if l > prev*.9999: print("Early stopping because there is no improvement") break prev = l # adjust the best result found so far for e,(l2,sc,ii) in enumerate(zip(l2s,scores,nimg)): if l2 < bestl2[e] and compare(sc, np.argmax(batchlab[e])): bestl2[e] = l2 bestscore[e] = np.argmax(sc) if l2 < o_bestl2[e] and compare(sc, np.argmax(batchlab[e])): o_bestl2[e] = l2 o_bestscore[e] = np.argmax(sc) o_bestattack[e] = ii o_best_const[e] = CONST[e] train_timer += time.time() - attack_begin_time # adjust the constant as needed for e in range(batch_size): if compare(bestscore[e], np.argmax(batchlab[e])) and bestscore[e] != -1: # modifier = self.sess.run(self.modifier) # np.save("best.model", modifier) print('old constant: ', CONST[e]) # success, divide const by two upper_bound[e] = min(upper_bound[e],CONST[e]) if upper_bound[e] < 1e9: CONST[e] = (lower_bound[e] + upper_bound[e])/2 print('new constant: ', CONST[e]) else: print('old constant: ', CONST[e]) # failure, either multiply by 10 if no solution found yet # or do binary search with the known upper bound lower_bound[e] = max(lower_bound[e],CONST[e]) if upper_bound[e] < 1e9: CONST[e] = (lower_bound[e] + upper_bound[e])/2 else: CONST[e] *= 10 print('new constant: ', CONST[e]) # return the best solution found o_bestl2 = np.array(o_bestl2) return np.array(o_bestattack), o_best_const ================================================ FILE: l2_attack_black.py ================================================ ## l2_attack_black.py -- attack a black-box network optimizing for l_2 distance ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2017, Huan Zhang . ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import sys import os import tensorflow as tf import numpy as np import scipy.misc from numba import jit import math import time BINARY_SEARCH_STEPS = 1 # number of times to adjust the constant with binary search MAX_ITERATIONS = 10000 # number of iterations to perform gradient descent ABORT_EARLY = True # if we stop improving, abort gradient descent early LEARNING_RATE = 2e-3 # larger values converge faster to less accurate results TARGETED = True # should we target one specific class? or just be wrong? CONFIDENCE = 0 # how strong the adversarial example should be INITIAL_CONST = 0.5 # the initial constant c to pick as a first guess @jit(nopython=True) def coordinate_ADAM(losses, indice, grad, hess, batch_size, mt_arr, vt_arr, real_modifier, up, down, lr, adam_epoch, beta1, beta2, proj): # indice = np.array(range(0, 3*299*299), dtype = np.int32) for i in range(batch_size): grad[i] = (losses[i*2+1] - losses[i*2+2]) / 0.0002 # true_grads = self.sess.run(self.grad_op, feed_dict={self.modifier: self.real_modifier}) # true_grads, losses, l2s, scores, nimgs = self.sess.run([self.grad_op, self.loss, self.l2dist, self.output, self.newimg], feed_dict={self.modifier: self.real_modifier}) # grad = true_grads[0].reshape(-1)[indice] # print(grad, true_grads[0].reshape(-1)[indice]) # self.real_modifier.reshape(-1)[indice] -= self.LEARNING_RATE * grad # self.real_modifier -= self.LEARNING_RATE * true_grads[0] # ADAM update mt = mt_arr[indice] mt = beta1 * mt + (1 - beta1) * grad mt_arr[indice] = mt vt = vt_arr[indice] vt = beta2 * vt + (1 - beta2) * (grad * grad) vt_arr[indice] = vt # epoch is an array; for each index we can have a different epoch number epoch = adam_epoch[indice] corr = (np.sqrt(1 - np.power(beta2,epoch))) / (1 - np.power(beta1, epoch)) m = real_modifier.reshape(-1) old_val = m[indice] old_val -= lr * corr * mt / (np.sqrt(vt) + 1e-8) # set it back to [-0.5, +0.5] region if proj: old_val = np.maximum(np.minimum(old_val, up[indice]), down[indice]) # print(grad) # print(old_val - m[indice]) m[indice] = old_val adam_epoch[indice] = epoch + 1 @jit(nopython=True) def coordinate_Newton(losses, indice, grad, hess, batch_size, mt_arr, vt_arr, real_modifier, up, down, lr, adam_epoch, beta1, beta2, proj): # def sign(x): # return np.piecewise(x, [x < 0, x >= 0], [-1, 1]) cur_loss = losses[0] for i in range(batch_size): grad[i] = (losses[i*2+1] - losses[i*2+2]) / 0.0002 hess[i] = (losses[i*2+1] - 2 * cur_loss + losses[i*2+2]) / (0.0001 * 0.0001) # print("New epoch:") # print('grad', grad) # print('hess', hess) # hess[hess < 0] = 1.0 # hess[np.abs(hess) < 0.1] = sign(hess[np.abs(hess) < 0.1]) * 0.1 # negative hessian cannot provide second order information, just do a gradient descent hess[hess < 0] = 1.0 # hessian too small, could be numerical problems hess[hess < 0.1] = 0.1 # print(hess) m = real_modifier.reshape(-1) old_val = m[indice] old_val -= lr * grad / hess # set it back to [-0.5, +0.5] region if proj: old_val = np.maximum(np.minimum(old_val, up[indice]), down[indice]) # print('delta', old_val - m[indice]) m[indice] = old_val # print(m[indice]) @jit(nopython=True) def coordinate_Newton_ADAM(losses, indice, grad, hess, batch_size, mt_arr, vt_arr, real_modifier, up, down, lr, adam_epoch, beta1, beta2, proj): cur_loss = losses[0] for i in range(batch_size): grad[i] = (losses[i*2+1] - losses[i*2+2]) / 0.0002 hess[i] = (losses[i*2+1] - 2 * cur_loss + losses[i*2+2]) / (0.0001 * 0.0001) # print("New epoch:") # print(grad) # print(hess) # positive hessian, using newton's method hess_indice = (hess >= 0) # print(hess_indice) # negative hessian, using ADAM adam_indice = (hess < 0) # print(adam_indice) # print(sum(hess_indice), sum(adam_indice)) hess[hess < 0] = 1.0 hess[hess < 0.1] = 0.1 # hess[np.abs(hess) < 0.1] = sign(hess[np.abs(hess) < 0.1]) * 0.1 # print(adam_indice) # Newton's Method m = real_modifier.reshape(-1) old_val = m[indice[hess_indice]] old_val -= lr * grad[hess_indice] / hess[hess_indice] # set it back to [-0.5, +0.5] region if proj: old_val = np.maximum(np.minimum(old_val, up[indice[hess_indice]]), down[indice[hess_indice]]) m[indice[hess_indice]] = old_val # ADMM mt = mt_arr[indice] mt = beta1 * mt + (1 - beta1) * grad mt_arr[indice] = mt vt = vt_arr[indice] vt = beta2 * vt + (1 - beta2) * (grad * grad) vt_arr[indice] = vt # epoch is an array; for each index we can have a different epoch number epoch = adam_epoch[indice] corr = (np.sqrt(1 - np.power(beta2,epoch[adam_indice]))) / (1 - np.power(beta1, epoch[adam_indice])) old_val = m[indice[adam_indice]] old_val -= lr * corr * mt[adam_indice] / (np.sqrt(vt[adam_indice]) + 1e-8) # old_val -= lr * grad[adam_indice] # set it back to [-0.5, +0.5] region if proj: old_val = np.maximum(np.minimum(old_val, up[indice[adam_indice]]), down[indice[adam_indice]]) m[indice[adam_indice]] = old_val adam_epoch[indice] = epoch + 1 # print(m[indice]) class BlackBoxL2: def __init__(self, sess, model, batch_size=1, confidence = CONFIDENCE, targeted = TARGETED, learning_rate = LEARNING_RATE, binary_search_steps = BINARY_SEARCH_STEPS, max_iterations = MAX_ITERATIONS, print_every = 100, early_stop_iters = 0, abort_early = ABORT_EARLY, initial_const = INITIAL_CONST, use_log = False, use_tanh = True, use_resize = False, adam_beta1 = 0.9, adam_beta2 = 0.999, reset_adam_after_found = False, solver = "adam", save_ckpts = "", load_checkpoint = "", start_iter = 0, init_size = 32, use_importance = True): """ The L_2 optimized attack. This attack is the most efficient and should be used as the primary attack to evaluate potential defenses. Returns adversarial examples for the supplied model. confidence: Confidence of adversarial examples: higher produces examples that are farther away, but more strongly classified as adversarial. batch_size: Number of gradient evaluations to run simultaneously. targeted: True if we should perform a targetted attack, False otherwise. learning_rate: The learning rate for the attack algorithm. Smaller values produce better results but are slower to converge. binary_search_steps: The number of times we perform binary search to find the optimal tradeoff-constant between distance and confidence. max_iterations: The maximum number of iterations. Larger values are more accurate; setting too small will require a large learning rate and will produce poor results. abort_early: If true, allows early aborts if gradient descent gets stuck. initial_const: The initial tradeoff-constant to use to tune the relative importance of distance and confidence. If binary_search_steps is large, the initial constant is not important. """ image_size, num_channels, num_labels = model.image_size, model.num_channels, model.num_labels self.model = model self.sess = sess self.TARGETED = targeted self.LEARNING_RATE = learning_rate self.MAX_ITERATIONS = max_iterations self.print_every = print_every self.early_stop_iters = early_stop_iters if early_stop_iters != 0 else max_iterations // 10 print("early stop:", self.early_stop_iters) self.BINARY_SEARCH_STEPS = binary_search_steps self.ABORT_EARLY = abort_early self.CONFIDENCE = confidence self.initial_const = initial_const self.start_iter = start_iter self.batch_size = batch_size self.num_channels = num_channels self.resize_init_size = init_size self.use_importance = use_importance if use_resize: self.small_x = self.resize_init_size self.small_y = self.resize_init_size else: self.small_x = image_size self.small_y = image_size self.use_tanh = use_tanh self.use_resize = use_resize self.save_ckpts = save_ckpts if save_ckpts: os.system("mkdir -p {}".format(save_ckpts)) self.repeat = binary_search_steps >= 10 # each batch has a different modifier value (see below) to evaluate # small_shape = (None,self.small_x,self.small_y,num_channels) shape = (None,image_size,image_size,num_channels) single_shape = (image_size, image_size, num_channels) small_single_shape = (self.small_x, self.small_y, num_channels) # the variable we're going to optimize over # support multiple batches # support any size image, will be resized to model native size if self.use_resize: self.modifier = tf.placeholder(tf.float32, shape=(None, None, None, None)) # scaled up image self.scaled_modifier = tf.image.resize_images(self.modifier, [image_size, image_size]) # operator used for resizing image self.resize_size_x = tf.placeholder(tf.int32) self.resize_size_y = tf.placeholder(tf.int32) self.resize_input = tf.placeholder(tf.float32, shape=(1, None, None, None)) self.resize_op = tf.image.resize_images(self.resize_input, [self.resize_size_x, self.resize_size_y]) else: self.modifier = tf.placeholder(tf.float32, shape=(None, image_size, image_size, num_channels)) # no resize self.scaled_modifier = self.modifier # the real variable, initialized to 0 self.load_checkpoint = load_checkpoint if load_checkpoint: # if checkpoint is incorrect reshape will fail print("Using checkpint", load_checkpoint) self.real_modifier = np.load(load_checkpoint).reshape((1,) + small_single_shape) else: self.real_modifier = np.zeros((1,) + small_single_shape, dtype=np.float32) # self.real_modifier = np.random.randn(image_size * image_size * num_channels).astype(np.float32).reshape((1,) + single_shape) # self.real_modifier /= np.linalg.norm(self.real_modifier) # these are variables to be more efficient in sending data to tf # we only work on 1 image at once; the batch is for evaluation loss at different modifiers self.timg = tf.Variable(np.zeros(single_shape), dtype=tf.float32) self.tlab = tf.Variable(np.zeros(num_labels), dtype=tf.float32) self.const = tf.Variable(0.0, dtype=tf.float32) # and here's what we use to assign them self.assign_timg = tf.placeholder(tf.float32, single_shape) self.assign_tlab = tf.placeholder(tf.float32, num_labels) self.assign_const = tf.placeholder(tf.float32) # the resulting image, tanh'd to keep bounded from -0.5 to 0.5 # broadcast self.timg to every dimension of modifier if use_tanh: self.newimg = tf.tanh(self.scaled_modifier + self.timg)/2 else: self.newimg = self.scaled_modifier + self.timg # prediction BEFORE-SOFTMAX of the model # now we have output at #batch_size different modifiers # the output should have shape (batch_size, num_labels) self.output = model.predict(self.newimg) # distance to the input data if use_tanh: self.l2dist = tf.reduce_sum(tf.square(self.newimg-tf.tanh(self.timg)/2), [1,2,3]) else: self.l2dist = tf.reduce_sum(tf.square(self.newimg - self.timg), [1,2,3]) # compute the probability of the label class versus the maximum other # self.tlab * self.output selects the Z value of real class # because self.tlab is an one-hot vector # the reduce_sum removes extra zeros, now get a vector of size #batch_size self.real = tf.reduce_sum((self.tlab)*self.output,1) # (1-self.tlab)*self.output gets all Z values for other classes # Because soft Z values are negative, it is possible that all Z values are less than 0 # and we mistakenly select the real class as the max. So we minus 10000 for real class self.other = tf.reduce_max((1-self.tlab)*self.output - (self.tlab*10000),1) # If self.targeted is true, then the targets represents the target labels. # If self.targeted is false, then targets are the original class labels. if self.TARGETED: if use_log: # loss1 = - tf.log(self.real) loss1 = tf.maximum(0.0, tf.log(self.other + 1e-30) - tf.log(self.real + 1e-30)) else: # if targetted, optimize for making the other class (real) most likely loss1 = tf.maximum(0.0, self.other-self.real+self.CONFIDENCE) else: if use_log: # loss1 = tf.log(self.real) loss1 = tf.maximum(0.0, tf.log(self.real + 1e-30) - tf.log(self.other + 1e-30)) else: # if untargeted, optimize for making this class least likely. loss1 = tf.maximum(0.0, self.real-self.other+self.CONFIDENCE) # sum up the losses (output is a vector of #batch_size) self.loss2 = self.l2dist self.loss1 = self.const*loss1 self.loss = self.loss1+self.loss2 # these are the variables to initialize when we run self.setup = [] self.setup.append(self.timg.assign(self.assign_timg)) self.setup.append(self.tlab.assign(self.assign_tlab)) self.setup.append(self.const.assign(self.assign_const)) # prepare the list of all valid variables var_size = self.small_x * self.small_y * num_channels self.use_var_len = var_size self.var_list = np.array(range(0, self.use_var_len), dtype = np.int32) self.used_var_list = np.zeros(var_size, dtype = np.int32) self.sample_prob = np.ones(var_size, dtype = np.float32) / var_size # upper and lower bounds for the modifier self.modifier_up = np.zeros(var_size, dtype = np.float32) self.modifier_down = np.zeros(var_size, dtype = np.float32) # random permutation for coordinate update self.perm = np.random.permutation(var_size) self.perm_index = 0 # ADAM status self.mt = np.zeros(var_size, dtype = np.float32) self.vt = np.zeros(var_size, dtype = np.float32) # self.beta1 = 0.8 # self.beta2 = 0.99 self.beta1 = adam_beta1 self.beta2 = adam_beta2 self.reset_adam_after_found = reset_adam_after_found self.adam_epoch = np.ones(var_size, dtype = np.int32) self.stage = 0 # variables used during optimization process self.grad = np.zeros(batch_size, dtype = np.float32) self.hess = np.zeros(batch_size, dtype = np.float32) # for testing self.grad_op = tf.gradients(self.loss, self.modifier) # compile numba function # self.coordinate_ADAM_numba = jit(coordinate_ADAM, nopython = True) # self.coordinate_ADAM_numba.recompile() # print(self.coordinate_ADAM_numba.inspect_llvm()) # np.set_printoptions(threshold=np.nan) # set solver solver = solver.lower() self.solver_name = solver if solver == "adam": self.solver = coordinate_ADAM elif solver == "newton": self.solver = coordinate_Newton elif solver == "adam_newton": self.solver = coordinate_Newton_ADAM elif solver != "fake_zero": print("unknown solver", solver) self.solver = coordinate_ADAM print("Using", solver, "solver") def max_pooling(self, image, size): img_pool = np.copy(image) img_x = image.shape[0] img_y = image.shape[1] for i in range(0, img_x, size): for j in range(0, img_y, size): img_pool[i:i+size, j:j+size] = np.max(image[i:i+size, j:j+size]) return img_pool def get_new_prob(self, prev_modifier, gen_double = False): prev_modifier = np.squeeze(prev_modifier) old_shape = prev_modifier.shape if gen_double: new_shape = (old_shape[0]*2, old_shape[1]*2, old_shape[2]) else: new_shape = old_shape prob = np.empty(shape=new_shape, dtype = np.float32) for i in range(prev_modifier.shape[2]): image = np.abs(prev_modifier[:,:,i]) image_pool = self.max_pooling(image, old_shape[0] // 8) if gen_double: prob[:,:,i] = scipy.misc.imresize(image_pool, 2.0, 'nearest', mode = 'F') else: prob[:,:,i] = image_pool prob /= np.sum(prob) return prob def resize_img(self, small_x, small_y, reset_only = False): self.small_x = small_x self.small_y = small_y small_single_shape = (self.small_x, self.small_y, self.num_channels) if reset_only: self.real_modifier = np.zeros((1,) + small_single_shape, dtype=np.float32) else: # run the resize_op once to get the scaled image prev_modifier = np.copy(self.real_modifier) self.real_modifier = self.sess.run(self.resize_op, feed_dict={self.resize_size_x: self.small_x, self.resize_size_y: self.small_y, self.resize_input: self.real_modifier}) # prepare the list of all valid variables var_size = self.small_x * self.small_y * self.num_channels self.use_var_len = var_size self.var_list = np.array(range(0, self.use_var_len), dtype = np.int32) # ADAM status self.mt = np.zeros(var_size, dtype = np.float32) self.vt = np.zeros(var_size, dtype = np.float32) self.adam_epoch = np.ones(var_size, dtype = np.int32) # update sample probability if reset_only: self.sample_prob = np.ones(var_size, dtype = np.float32) / var_size else: self.sample_prob = self.get_new_prob(prev_modifier, True) self.sample_prob = self.sample_prob.reshape(var_size) def fake_blackbox_optimizer(self): true_grads, losses, l2s, loss1, loss2, scores, nimgs = self.sess.run([self.grad_op, self.loss, self.l2dist, self.loss1, self.loss2, self.output, self.newimg], feed_dict={self.modifier: self.real_modifier}) # ADAM update grad = true_grads[0].reshape(-1) # print(true_grads[0]) epoch = self.adam_epoch[0] mt = self.beta1 * self.mt + (1 - self.beta1) * grad vt = self.beta2 * self.vt + (1 - self.beta2) * np.square(grad) corr = (math.sqrt(1 - self.beta2 ** epoch)) / (1 - self.beta1 ** epoch) # print(grad.shape, mt.shape, vt.shape, self.real_modifier.shape) # m is a *view* of self.real_modifier m = self.real_modifier.reshape(-1) # this is in-place m -= self.LEARNING_RATE * corr * (mt / (np.sqrt(vt) + 1e-8)) self.mt = mt self.vt = vt # m -= self.LEARNING_RATE * grad if not self.use_tanh: m_proj = np.maximum(np.minimum(m, self.modifier_up), self.modifier_down) np.copyto(m, m_proj) self.adam_epoch[0] = epoch + 1 return losses[0], l2s[0], loss1[0], loss2[0], scores[0], nimgs[0] def blackbox_optimizer(self, iteration): # build new inputs, based on current variable value var = np.repeat(self.real_modifier, self.batch_size * 2 + 1, axis=0) var_size = self.real_modifier.size # print(s, "variables remaining") # var_indice = np.random.randint(0, self.var_list.size, size=self.batch_size) if self.use_importance: var_indice = np.random.choice(self.var_list.size, self.batch_size, replace=False, p = self.sample_prob) else: var_indice = np.random.choice(self.var_list.size, self.batch_size, replace=False) indice = self.var_list[var_indice] # indice = self.var_list # regenerate the permutations if we run out # if self.perm_index + self.batch_size >= var_size: # self.perm = np.random.permutation(var_size) # self.perm_index = 0 # indice = self.perm[self.perm_index:self.perm_index + self.batch_size] # b[0] has the original modifier, b[1] has one index added 0.0001 for i in range(self.batch_size): var[i * 2 + 1].reshape(-1)[indice[i]] += 0.0001 var[i * 2 + 2].reshape(-1)[indice[i]] -= 0.0001 losses, l2s, loss1, loss2, scores, nimgs = self.sess.run([self.loss, self.l2dist, self.loss1, self.loss2, self.output, self.newimg], feed_dict={self.modifier: var}) # losses = self.sess.run(self.loss, feed_dict={self.modifier: var}) # t_grad = self.sess.run(self.grad_op, feed_dict={self.modifier: self.real_modifier}) # self.grad = t_grad[0].reshape(-1) # true_grads = self.sess.run(self.grad_op, feed_dict={self.modifier: self.real_modifier}) # self.coordinate_ADAM_numba(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh) # coordinate_ADAM(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh) # coordinate_ADAM(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh, true_grads) # coordinate_Newton(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh) # coordinate_Newton_ADAM(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh) self.solver(losses, indice, self.grad, self.hess, self.batch_size, self.mt, self.vt, self.real_modifier, self.modifier_up, self.modifier_down, self.LEARNING_RATE, self.adam_epoch, self.beta1, self.beta2, not self.use_tanh) # adjust sample probability, sample around the points with large gradient if self.save_ckpts: np.save('{}/iter{}'.format(self.save_ckpts, iteration), self.real_modifier) if self.real_modifier.shape[0] > self.resize_init_size: self.sample_prob = self.get_new_prob(self.real_modifier) # self.sample_prob = self.get_new_prob(tmp_mt.reshape(self.real_modifier.shape)) self.sample_prob = self.sample_prob.reshape(var_size) # if the gradient is too small, do not optimize on this variable # self.var_list = np.delete(self.var_list, indice[np.abs(self.grad) < 5e-3]) # reset the list every 10000 iterations # if iteration%200 == 0: # print("{} variables remained at last stage".format(self.var_list.size)) # var_size = self.real_modifier.size # self.var_list = np.array(range(0, var_size)) return losses[0], l2s[0], loss1[0], loss2[0], scores[0], nimgs[0] # return losses[0] def attack(self, imgs, targets): """ Perform the L_2 attack on the given images for the given targets. If self.targeted is true, then the targets represents the target labels. If self.targeted is false, then targets are the original class labels. """ r = [] print('go up to',len(imgs)) # we can only run 1 image at a time, minibatches are used for gradient evaluation for i in range(0,len(imgs)): print('tick',i) r.extend(self.attack_batch(imgs[i], targets[i])) return np.array(r) # only accepts 1 image at a time. Batch is used for gradient evaluations at different points def attack_batch(self, img, lab): """ Run the attack on a batch of images and labels. """ def compare(x,y): if not isinstance(x, (float, int, np.int64)): x = np.copy(x) if self.TARGETED: x[y] -= self.CONFIDENCE else: x[y] += self.CONFIDENCE x = np.argmax(x) if self.TARGETED: return x == y else: return x != y # remove the extra batch dimension if len(img.shape) == 4: img = img[0] if len(lab.shape) == 2: lab = lab[0] # convert to tanh-space if self.use_tanh: img = np.arctanh(img*1.999999) # set the lower and upper bounds accordingly lower_bound = 0.0 CONST = self.initial_const upper_bound = 1e10 # convert img to float32 to avoid numba error img = img.astype(np.float32) # set the upper and lower bounds for the modifier if not self.use_tanh: self.modifier_up = 0.5 - img.reshape(-1) self.modifier_down = -0.5 - img.reshape(-1) # clear the modifier if not self.load_checkpoint: if self.use_resize: self.resize_img(self.resize_init_size, self.resize_init_size, True) else: self.real_modifier.fill(0.0) # the best l2, score, and image attack o_best_const = CONST o_bestl2 = 1e10 o_bestscore = -1 o_bestattack = img for outer_step in range(self.BINARY_SEARCH_STEPS): print(o_bestl2) bestl2 = 1e10 bestscore = -1 # The last iteration (if we run many steps) repeat the search once. if self.repeat == True and outer_step == self.BINARY_SEARCH_STEPS-1: CONST = upper_bound # set the variables so that we don't have to send them over again self.sess.run(self.setup, {self.assign_timg: img, self.assign_tlab: lab, self.assign_const: CONST}) # use the current best model # np.copyto(self.real_modifier, o_bestattack - img) # use the model left by last constant change prev = 1e6 train_timer = 0.0 last_loss1 = 1.0 if not self.load_checkpoint: if self.use_resize: self.resize_img(self.resize_init_size, self.resize_init_size, True) else: self.real_modifier.fill(0.0) # reset ADAM status self.mt.fill(0.0) self.vt.fill(0.0) self.adam_epoch.fill(1) self.stage = 0 multiplier = 1 eval_costs = 0 if self.solver_name != "fake_zero": multiplier = 24 for iteration in range(self.start_iter, self.MAX_ITERATIONS): if self.use_resize: if iteration == 2000: # if iteration == 2000 // 24: self.resize_img(64,64) if iteration == 10000: # if iteration == 2000 // 24 + (10000 - 2000) // 96: self.resize_img(128,128) # if iteration == 200*30: # if iteration == 250 * multiplier: # self.resize_img(256,256) # print out the losses every 10% if iteration%(self.print_every) == 0: # print(iteration,self.sess.run((self.loss,self.real,self.other,self.loss1,self.loss2), feed_dict={self.modifier: self.real_modifier})) loss, real, other, loss1, loss2 = self.sess.run((self.loss,self.real,self.other,self.loss1,self.loss2), feed_dict={self.modifier: self.real_modifier}) print("[STATS][L2] iter = {}, cost = {}, time = {:.3f}, size = {}, loss = {:.5g}, real = {:.5g}, other = {:.5g}, loss1 = {:.5g}, loss2 = {:.5g}".format(iteration, eval_costs, train_timer, self.real_modifier.shape, loss[0], real[0], other[0], loss1[0], loss2[0])) sys.stdout.flush() # np.save('black_iter_{}'.format(iteration), self.real_modifier) attack_begin_time = time.time() # perform the attack if self.solver_name == "fake_zero": l, l2, loss1, loss2, score, nimg = self.fake_blackbox_optimizer() else: l, l2, loss1, loss2, score, nimg = self.blackbox_optimizer(iteration) # l = self.blackbox_optimizer(iteration) if self.solver_name == "fake_zero": eval_costs += np.prod(self.real_modifier.shape) else: eval_costs += self.batch_size # reset ADAM states when a valid example has been found if loss1 == 0.0 and last_loss1 != 0.0 and self.stage == 0: # we have reached the fine tunning point # reset ADAM to avoid overshoot if self.reset_adam_after_found: self.mt.fill(0.0) self.vt.fill(0.0) self.adam_epoch.fill(1) self.stage = 1 last_loss1 = loss1 # check if we should abort search if we're getting nowhere. # if self.ABORT_EARLY and iteration%(self.MAX_ITERATIONS//10) == 0: if self.ABORT_EARLY and iteration % self.early_stop_iters == 0: if l > prev*.9999: print("Early stopping because there is no improvement") break prev = l # adjust the best result found so far # the best attack should have the target class with the largest value, # and has smallest l2 distance if l2 < bestl2 and compare(score, np.argmax(lab)): bestl2 = l2 bestscore = np.argmax(score) if l2 < o_bestl2 and compare(score, np.argmax(lab)): # print a message if it is the first attack found if o_bestl2 == 1e10: print("[STATS][L3](First valid attack found!) iter = {}, cost = {}, time = {:.3f}, size = {}, loss = {:.5g}, loss1 = {:.5g}, loss2 = {:.5g}, l2 = {:.5g}".format(iteration, eval_costs, train_timer, self.real_modifier.shape, l, loss1, loss2, l2)) sys.stdout.flush() o_bestl2 = l2 o_bestscore = np.argmax(score) o_bestattack = nimg o_best_const = CONST train_timer += time.time() - attack_begin_time # adjust the constant as needed if compare(bestscore, np.argmax(lab)) and bestscore != -1: # success, divide const by two print('old constant: ', CONST) upper_bound = min(upper_bound,CONST) if upper_bound < 1e9: CONST = (lower_bound + upper_bound)/2 print('new constant: ', CONST) else: # failure, either multiply by 10 if no solution found yet # or do binary search with the known upper bound print('old constant: ', CONST) lower_bound = max(lower_bound,CONST) if upper_bound < 1e9: CONST = (lower_bound + upper_bound)/2 else: CONST *= 10 print('new constant: ', CONST) # return the best solution found return o_bestattack, o_best_const ================================================ FILE: labels/imagenet_val_to_carlini.py ================================================ #!/usr/bin/env python3 import os import sys import glob f = open('label2num.txt') mapping = {} for line in f: l = line.strip().split(':') mapping[l[1]] = l[0] print("Total {} classes loaded".format(len(mapping))) os.system("mkdir -p imgs") file_list = glob.glob('val/**/*.JPEG', recursive=True) print("Total {} files found".format(len(file_list))) cur = 1 total = len(file_list) for img_path in file_list: s = img_path.split('/')[1] n = os.path.splitext(os.path.basename(img_path))[0].split('_')[2] label = mapping[s] os.system("cp {} imgs/{}.{}.jpg".format(img_path, label, n)) if cur % 1000 == 0: print("{}/{} finished ".format(cur, total)) cur += 1 print() ================================================ FILE: labels/label2num.txt ================================================ 1:n01440764:tench, Tinca tinca 2:n01443537:goldfish, Carassius auratus 3:n01484850:great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias 4:n01491361:tiger shark, Galeocerdo cuvieri 5:n01494475:hammerhead, hammerhead shark 6:n01496331:electric ray, crampfish, numbfish, torpedo 7:n01498041:stingray 8:n01514668:cock 9:n01514859:hen 10:n01518878:ostrich, Struthio camelus 11:n01530575:brambling, Fringilla montifringilla 12:n01531178:goldfinch, Carduelis carduelis 13:n01532829:house finch, linnet, Carpodacus mexicanus 14:n01534433:junco, snowbird 15:n01537544:indigo bunting, indigo finch, indigo bird, Passerina cyanea 16:n01558993:robin, American robin, Turdus migratorius 17:n01560419:bulbul 18:n01580077:jay 19:n01582220:magpie 20:n01592084:chickadee 21:n01601694:water ouzel, dipper 22:n01608432:kite 23:n01614925:bald eagle, American eagle, Haliaeetus leucocephalus 24:n01616318:vulture 25:n01622779:great grey owl, great gray owl, Strix nebulosa 26:n01629819:European fire salamander, Salamandra salamandra 27:n01630670:common newt, Triturus vulgaris 28:n01631663:eft 29:n01632458:spotted salamander, Ambystoma maculatum 30:n01632777:axolotl, mud puppy, Ambystoma mexicanum 31:n01641577:bullfrog, Rana catesbeiana 32:n01644373:tree frog, tree-frog 33:n01644900:tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui 34:n01664065:loggerhead, loggerhead turtle, Caretta caretta 35:n01665541:leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea 36:n01667114:mud turtle 37:n01667778:terrapin 38:n01669191:box turtle, box tortoise 39:n01675722:banded gecko 40:n01677366:common iguana, iguana, Iguana iguana 41:n01682714:American chameleon, anole, Anolis carolinensis 42:n01685808:whiptail, whiptail lizard 43:n01687978:agama 44:n01688243:frilled lizard, Chlamydosaurus kingi 45:n01689811:alligator lizard 46:n01692333:Gila monster, Heloderma suspectum 47:n01693334:green lizard, Lacerta viridis 48:n01694178:African chameleon, Chamaeleo chamaeleon 49:n01695060:Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis 50:n01697457:African crocodile, Nile crocodile, Crocodylus niloticus 51:n01698640:American alligator, Alligator mississipiensis 52:n01704323:triceratops 53:n01728572:thunder snake, worm snake, Carphophis amoenus 54:n01728920:ringneck snake, ring-necked snake, ring snake 55:n01729322:hognose snake, puff adder, sand viper 56:n01729977:green snake, grass snake 57:n01734418:king snake, kingsnake 58:n01735189:garter snake, grass snake 59:n01737021:water snake 60:n01739381:vine snake 61:n01740131:night snake, Hypsiglena torquata 62:n01742172:boa constrictor, Constrictor constrictor 63:n01744401:rock python, rock snake, Python sebae 64:n01748264:Indian cobra, Naja naja 65:n01749939:green mamba 66:n01751748:sea snake 67:n01753488:horned viper, cerastes, sand viper, horned asp, Cerastes cornutus 68:n01755581:diamondback, diamondback rattlesnake, Crotalus adamanteus 69:n01756291:sidewinder, horned rattlesnake, Crotalus cerastes 70:n01768244:trilobite 71:n01770081:harvestman, daddy longlegs, Phalangium opilio 72:n01770393:scorpion 73:n01773157:black and gold garden spider, Argiope aurantia 74:n01773549:barn spider, Araneus cavaticus 75:n01773797:garden spider, Aranea diademata 76:n01774384:black widow, Latrodectus mactans 77:n01774750:tarantula 78:n01775062:wolf spider, hunting spider 79:n01776313:tick 80:n01784675:centipede 81:n01795545:black grouse 82:n01796340:ptarmigan 83:n01797886:ruffed grouse, partridge, Bonasa umbellus 84:n01798484:prairie chicken, prairie grouse, prairie fowl 85:n01806143:peacock 86:n01806567:quail 87:n01807496:partridge 88:n01817953:African grey, African gray, Psittacus erithacus 89:n01818515:macaw 90:n01819313:sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita 91:n01820546:lorikeet 92:n01824575:coucal 93:n01828970:bee eater 94:n01829413:hornbill 95:n01833805:hummingbird 96:n01843065:jacamar 97:n01843383:toucan 98:n01847000:drake 99:n01855032:red-breasted merganser, Mergus serrator 100:n01855672:goose 101:n01860187:black swan, Cygnus atratus 102:n01871265:tusker 103:n01872401:echidna, spiny anteater, anteater 104:n01873310:platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus 105:n01877812:wallaby, brush kangaroo 106:n01882714:koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus 107:n01883070:wombat 108:n01910747:jellyfish 109:n01914609:sea anemone, anemone 110:n01917289:brain coral 111:n01924916:flatworm, platyhelminth 112:n01930112:nematode, nematode worm, roundworm 113:n01943899:conch 114:n01944390:snail 115:n01945685:slug 116:n01950731:sea slug, nudibranch 117:n01955084:chiton, coat-of-mail shell, sea cradle, polyplacophore 118:n01968897:chambered nautilus, pearly nautilus, nautilus 119:n01978287:Dungeness crab, Cancer magister 120:n01978455:rock crab, Cancer irroratus 121:n01980166:fiddler crab 122:n01981276:king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica 123:n01983481:American lobster, Northern lobster, Maine lobster, Homarus americanus 124:n01984695:spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish 125:n01985128:crayfish, crawfish, crawdad, crawdaddy 126:n01986214:hermit crab 127:n01990800:isopod 128:n02002556:white stork, Ciconia ciconia 129:n02002724:black stork, Ciconia nigra 130:n02006656:spoonbill 131:n02007558:flamingo 132:n02009229:little blue heron, Egretta caerulea 133:n02009912:American egret, great white heron, Egretta albus 134:n02011460:bittern 518:n02012849:crane 136:n02013706:limpkin, Aramus pictus 137:n02017213:European gallinule, Porphyrio porphyrio 138:n02018207:American coot, marsh hen, mud hen, water hen, Fulica americana 139:n02018795:bustard 140:n02025239:ruddy turnstone, Arenaria interpres 141:n02027492:red-backed sandpiper, dunlin, Erolia alpina 142:n02028035:redshank, Tringa totanus 143:n02033041:dowitcher 144:n02037110:oystercatcher, oyster catcher 145:n02051845:pelican 146:n02056570:king penguin, Aptenodytes patagonica 147:n02058221:albatross, mollymawk 148:n02066245:grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus 149:n02071294:killer whale, killer, orca, grampus, sea wolf, Orcinus orca 150:n02074367:dugong, Dugong dugon 151:n02077923:sea lion 152:n02085620:Chihuahua 153:n02085782:Japanese spaniel 154:n02085936:Maltese dog, Maltese terrier, Maltese 155:n02086079:Pekinese, Pekingese, Peke 156:n02086240:Shih-Tzu 157:n02086646:Blenheim spaniel 158:n02086910:papillon 159:n02087046:toy terrier 160:n02087394:Rhodesian ridgeback 161:n02088094:Afghan hound, Afghan 162:n02088238:basset, basset hound 163:n02088364:beagle 164:n02088466:bloodhound, sleuthhound 165:n02088632:bluetick 166:n02089078:black-and-tan coonhound 167:n02089867:Walker hound, Walker foxhound 168:n02089973:English foxhound 169:n02090379:redbone 170:n02090622:borzoi, Russian wolfhound 171:n02090721:Irish wolfhound 172:n02091032:Italian greyhound 173:n02091134:whippet 174:n02091244:Ibizan hound, Ibizan Podenco 175:n02091467:Norwegian elkhound, elkhound 176:n02091635:otterhound, otter hound 177:n02091831:Saluki, gazelle hound 178:n02092002:Scottish deerhound, deerhound 179:n02092339:Weimaraner 180:n02093256:Staffordshire bullterrier, Staffordshire bull terrier 181:n02093428:American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier 182:n02093647:Bedlington terrier 183:n02093754:Border terrier 184:n02093859:Kerry blue terrier 185:n02093991:Irish terrier 186:n02094114:Norfolk terrier 187:n02094258:Norwich terrier 188:n02094433:Yorkshire terrier 189:n02095314:wire-haired fox terrier 190:n02095570:Lakeland terrier 191:n02095889:Sealyham terrier, Sealyham 192:n02096051:Airedale, Airedale terrier 193:n02096177:cairn, cairn terrier 194:n02096294:Australian terrier 195:n02096437:Dandie Dinmont, Dandie Dinmont terrier 196:n02096585:Boston bull, Boston terrier 197:n02097047:miniature schnauzer 198:n02097130:giant schnauzer 199:n02097209:standard schnauzer 200:n02097298:Scotch terrier, Scottish terrier, Scottie 201:n02097474:Tibetan terrier, chrysanthemum dog 202:n02097658:silky terrier, Sydney silky 203:n02098105:soft-coated wheaten terrier 204:n02098286:West Highland white terrier 205:n02098413:Lhasa, Lhasa apso 206:n02099267:flat-coated retriever 207:n02099429:curly-coated retriever 208:n02099601:golden retriever 209:n02099712:Labrador retriever 210:n02099849:Chesapeake Bay retriever 211:n02100236:German short-haired pointer 212:n02100583:vizsla, Hungarian pointer 213:n02100735:English setter 214:n02100877:Irish setter, red setter 215:n02101006:Gordon setter 216:n02101388:Brittany spaniel 217:n02101556:clumber, clumber spaniel 218:n02102040:English springer, English springer spaniel 219:n02102177:Welsh springer spaniel 220:n02102318:cocker spaniel, English cocker spaniel, cocker 221:n02102480:Sussex spaniel 222:n02102973:Irish water spaniel 223:n02104029:kuvasz 224:n02104365:schipperke 225:n02105056:groenendael 226:n02105162:malinois 227:n02105251:briard 228:n02105412:kelpie 229:n02105505:komondor 230:n02105641:Old English sheepdog, bobtail 231:n02105855:Shetland sheepdog, Shetland sheep dog, Shetland 232:n02106030:collie 233:n02106166:Border collie 234:n02106382:Bouvier des Flandres, Bouviers des Flandres 235:n02106550:Rottweiler 236:n02106662:German shepherd, German shepherd dog, German police dog, alsatian 237:n02107142:Doberman, Doberman pinscher 238:n02107312:miniature pinscher 239:n02107574:Greater Swiss Mountain dog 240:n02107683:Bernese mountain dog 241:n02107908:Appenzeller 242:n02108000:EntleBucher 243:n02108089:boxer 244:n02108422:bull mastiff 245:n02108551:Tibetan mastiff 246:n02108915:French bulldog 247:n02109047:Great Dane 248:n02109525:Saint Bernard, St Bernard 249:n02109961:Eskimo dog, husky 250:n02110063:malamute, malemute, Alaskan malamute 251:n02110185:Siberian husky 252:n02110341:dalmatian, coach dog, carriage dog 253:n02110627:affenpinscher, monkey pinscher, monkey dog 254:n02110806:basenji 255:n02110958:pug, pug-dog 256:n02111129:Leonberg 257:n02111277:Newfoundland, Newfoundland dog 258:n02111500:Great Pyrenees 259:n02111889:Samoyed, Samoyede 260:n02112018:Pomeranian 261:n02112137:chow, chow chow 262:n02112350:keeshond 263:n02112706:Brabancon griffon 264:n02113023:Pembroke, Pembroke Welsh corgi 265:n02113186:Cardigan, Cardigan Welsh corgi 266:n02113624:toy poodle 267:n02113712:miniature poodle 268:n02113799:standard poodle 269:n02113978:Mexican hairless 270:n02114367:timber wolf, grey wolf, gray wolf, Canis lupus 271:n02114548:white wolf, Arctic wolf, Canis lupus tundrarum 272:n02114712:red wolf, maned wolf, Canis rufus, Canis niger 273:n02114855:coyote, prairie wolf, brush wolf, Canis latrans 274:n02115641:dingo, warrigal, warragal, Canis dingo 275:n02115913:dhole, Cuon alpinus 276:n02116738:African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus 277:n02117135:hyena, hyaena 278:n02119022:red fox, Vulpes vulpes 279:n02119789:kit fox, Vulpes macrotis 280:n02120079:Arctic fox, white fox, Alopex lagopus 281:n02120505:grey fox, gray fox, Urocyon cinereoargenteus 282:n02123045:tabby, tabby cat 283:n02123159:tiger cat 284:n02123394:Persian cat 285:n02123597:Siamese cat, Siamese 286:n02124075:Egyptian cat 287:n02125311:cougar, puma, catamount, mountain lion, painter, panther, Felis concolor 288:n02127052:lynx, catamount 289:n02128385:leopard, Panthera pardus 290:n02128757:snow leopard, ounce, Panthera uncia 291:n02128925:jaguar, panther, Panthera onca, Felis onca 292:n02129165:lion, king of beasts, Panthera leo 293:n02129604:tiger, Panthera tigris 294:n02130308:cheetah, chetah, Acinonyx jubatus 295:n02132136:brown bear, bruin, Ursus arctos 296:n02133161:American black bear, black bear, Ursus americanus, Euarctos americanus 297:n02134084:ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus 298:n02134418:sloth bear, Melursus ursinus, Ursus ursinus 299:n02137549:mongoose 300:n02138441:meerkat, mierkat 301:n02165105:tiger beetle 302:n02165456:ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle 303:n02167151:ground beetle, carabid beetle 304:n02168699:long-horned beetle, longicorn, longicorn beetle 305:n02169497:leaf beetle, chrysomelid 306:n02172182:dung beetle 307:n02174001:rhinoceros beetle 308:n02177972:weevil 309:n02190166:fly 310:n02206856:bee 311:n02219486:ant, emmet, pismire 312:n02226429:grasshopper, hopper 313:n02229544:cricket 314:n02231487:walking stick, walkingstick, stick insect 315:n02233338:cockroach, roach 316:n02236044:mantis, mantid 317:n02256656:cicada, cicala 318:n02259212:leafhopper 319:n02264363:lacewing, lacewing fly 320:n02268443:dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk 321:n02268853:damselfly 322:n02276258:admiral 323:n02277742:ringlet, ringlet butterfly 324:n02279972:monarch, monarch butterfly, milkweed butterfly, Danaus plexippus 325:n02280649:cabbage butterfly 326:n02281406:sulphur butterfly, sulfur butterfly 327:n02281787:lycaenid, lycaenid butterfly 328:n02317335:starfish, sea star 329:n02319095:sea urchin 330:n02321529:sea cucumber, holothurian 331:n02325366:wood rabbit, cottontail, cottontail rabbit 332:n02326432:hare 333:n02328150:Angora, Angora rabbit 334:n02342885:hamster 335:n02346627:porcupine, hedgehog 336:n02356798:fox squirrel, eastern fox squirrel, Sciurus niger 337:n02361337:marmot 338:n02363005:beaver 339:n02364673:guinea pig, Cavia cobaya 340:n02389026:sorrel 341:n02391049:zebra 342:n02395406:hog, pig, grunter, squealer, Sus scrofa 343:n02396427:wild boar, boar, Sus scrofa 344:n02397096:warthog 345:n02398521:hippopotamus, hippo, river horse, Hippopotamus amphibius 346:n02403003:ox 347:n02408429:water buffalo, water ox, Asiatic buffalo, Bubalus bubalis 348:n02410509:bison 349:n02412080:ram, tup 350:n02415577:bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis 351:n02417914:ibex, Capra ibex 352:n02422106:hartebeest 353:n02422699:impala, Aepyceros melampus 354:n02423022:gazelle 355:n02437312:Arabian camel, dromedary, Camelus dromedarius 356:n02437616:llama 357:n02441942:weasel 358:n02442845:mink 359:n02443114:polecat, fitch, foulmart, foumart, Mustela putorius 360:n02443484:black-footed ferret, ferret, Mustela nigripes 361:n02444819:otter 362:n02445715:skunk, polecat, wood pussy 363:n02447366:badger 364:n02454379:armadillo 365:n02457408:three-toed sloth, ai, Bradypus tridactylus 366:n02480495:orangutan, orang, orangutang, Pongo pygmaeus 367:n02480855:gorilla, Gorilla gorilla 368:n02481823:chimpanzee, chimp, Pan troglodytes 369:n02483362:gibbon, Hylobates lar 370:n02483708:siamang, Hylobates syndactylus, Symphalangus syndactylus 371:n02484975:guenon, guenon monkey 372:n02486261:patas, hussar monkey, Erythrocebus patas 373:n02486410:baboon 374:n02487347:macaque 375:n02488291:langur 376:n02488702:colobus, colobus monkey 377:n02489166:proboscis monkey, Nasalis larvatus 378:n02490219:marmoset 379:n02492035:capuchin, ringtail, Cebus capucinus 380:n02492660:howler monkey, howler 381:n02493509:titi, titi monkey 382:n02493793:spider monkey, Ateles geoffroyi 383:n02494079:squirrel monkey, Saimiri sciureus 384:n02497673:Madagascar cat, ring-tailed lemur, Lemur catta 385:n02500267:indri, indris, Indri indri, Indri brevicaudatus 386:n02504013:Indian elephant, Elephas maximus 387:n02504458:African elephant, Loxodonta africana 388:n02509815:lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens 389:n02510455:giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca 390:n02514041:barracouta, snoek 391:n02526121:eel 392:n02536864:coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch 393:n02606052:rock beauty, Holocanthus tricolor 394:n02607072:anemone fish 395:n02640242:sturgeon 396:n02641379:gar, garfish, garpike, billfish, Lepisosteus osseus 397:n02643566:lionfish 398:n02655020:puffer, pufferfish, blowfish, globefish 399:n02666196:abacus 400:n02667093:abaya 401:n02669723:academic gown, academic robe, judge's robe 402:n02672831:accordion, piano accordion, squeeze box 403:n02676566:acoustic guitar 404:n02687172:aircraft carrier, carrier, flattop, attack aircraft carrier 405:n02690373:airliner 406:n02692877:airship, dirigible 407:n02699494:altar 408:n02701002:ambulance 409:n02704792:amphibian, amphibious vehicle 410:n02708093:analog clock 411:n02727426:apiary, bee house 412:n02730930:apron 413:n02747177:ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin 414:n02749479:assault rifle, assault gun 415:n02769748:backpack, back pack, knapsack, packsack, rucksack, haversack 416:n02776631:bakery, bakeshop, bakehouse 417:n02777292:balance beam, beam 418:n02782093:balloon 419:n02783161:ballpoint, ballpoint pen, ballpen, Biro 420:n02786058:Band Aid 421:n02787622:banjo 422:n02788148:bannister, banister, balustrade, balusters, handrail 423:n02790996:barbell 424:n02791124:barber chair 425:n02791270:barbershop 426:n02793495:barn 427:n02794156:barometer 428:n02795169:barrel, cask 429:n02797295:barrow, garden cart, lawn cart, wheelbarrow 430:n02799071:baseball 431:n02802426:basketball 432:n02804414:bassinet 433:n02804610:bassoon 434:n02807133:bathing cap, swimming cap 435:n02808304:bath towel 436:n02808440:bathtub, bathing tub, bath, tub 437:n02814533:beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon 438:n02814860:beacon, lighthouse, beacon light, pharos 439:n02815834:beaker 440:n02817516:bearskin, busby, shako 441:n02823428:beer bottle 442:n02823750:beer glass 443:n02825657:bell cote, bell cot 444:n02834397:bib 445:n02835271:bicycle-built-for-two, tandem bicycle, tandem 446:n02837789:bikini, two-piece 447:n02840245:binder, ring-binder 448:n02841315:binoculars, field glasses, opera glasses 449:n02843684:birdhouse 450:n02859443:boathouse 451:n02860847:bobsled, bobsleigh, bob 452:n02865351:bolo tie, bolo, bola tie, bola 453:n02869837:bonnet, poke bonnet 454:n02870880:bookcase 455:n02871525:bookshop, bookstore, bookstall 456:n02877765:bottlecap 457:n02879718:bow 458:n02883205:bow tie, bow-tie, bowtie 459:n02892201:brass, memorial tablet, plaque 460:n02892767:brassiere, bra, bandeau 461:n02894605:breakwater, groin, groyne, mole, bulwark, seawall, jetty 462:n02895154:breastplate, aegis, egis 463:n02906734:broom 464:n02909870:bucket, pail 465:n02910353:buckle 466:n02916936:bulletproof vest 467:n02917067:bullet train, bullet 468:n02927161:butcher shop, meat market 469:n02930766:cab, hack, taxi, taxicab 470:n02939185:caldron, cauldron 471:n02948072:candle, taper, wax light 472:n02950826:cannon 473:n02951358:canoe 474:n02951585:can opener, tin opener 475:n02963159:cardigan 476:n02965783:car mirror 477:n02966193:carousel, carrousel, merry-go-round, roundabout, whirligig 478:n02966687:carpenter's kit, tool kit 479:n02971356:carton 480:n02974003:car wheel 481:n02977058:cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM 482:n02978881:cassette 483:n02979186:cassette player 484:n02980441:castle 485:n02981792:catamaran 486:n02988304:CD player 487:n02992211:cello, violoncello 488:n02992529:cellular telephone, cellular phone, cellphone, cell, mobile phone 489:n02999410:chain 490:n03000134:chainlink fence 491:n03000247:chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour 492:n03000684:chain saw, chainsaw 493:n03014705:chest 494:n03016953:chiffonier, commode 495:n03017168:chime, bell, gong 496:n03018349:china cabinet, china closet 497:n03026506:Christmas stocking 498:n03028079:church, church building 499:n03032252:cinema, movie theater, movie theatre, movie house, picture palace 500:n03041632:cleaver, meat cleaver, chopper 501:n03042490:cliff dwelling 502:n03045698:cloak 503:n03047690:clog, geta, patten, sabot 504:n03062245:cocktail shaker 505:n03063599:coffee mug 506:n03063689:coffeepot 507:n03065424:coil, spiral, volute, whorl, helix 508:n03075370:combination lock 509:n03085013:computer keyboard, keypad 510:n03089624:confectionery, confectionary, candy store 511:n03095699:container ship, containership, container vessel 512:n03100240:convertible 513:n03109150:corkscrew, bottle screw 514:n03110669:cornet, horn, trumpet, trump 515:n03124043:cowboy boot 516:n03124170:cowboy hat, ten-gallon hat 517:n03125729:cradle 518:n03126707:crane 519:n03127747:crash helmet 520:n03127925:crate 521:n03131574:crib, cot 522:n03133878:Crock Pot 523:n03134739:croquet ball 524:n03141823:crutch 525:n03146219:cuirass 526:n03160309:dam, dike, dyke 527:n03179701:desk 528:n03180011:desktop computer 529:n03187595:dial telephone, dial phone 530:n03188531:diaper, nappy, napkin 531:n03196217:digital clock 532:n03197337:digital watch 533:n03201208:dining table, board 534:n03207743:dishrag, dishcloth 535:n03207941:dishwasher, dish washer, dishwashing machine 536:n03208938:disk brake, disc brake 537:n03216828:dock, dockage, docking facility 538:n03218198:dogsled, dog sled, dog sleigh 539:n03220513:dome 540:n03223299:doormat, welcome mat 541:n03240683:drilling platform, offshore rig 542:n03249569:drum, membranophone, tympan 543:n03250847:drumstick 544:n03255030:dumbbell 545:n03259280:Dutch oven 546:n03271574:electric fan, blower 547:n03272010:electric guitar 548:n03272562:electric locomotive 549:n03290653:entertainment center 550:n03291819:envelope 551:n03297495:espresso maker 552:n03314780:face powder 553:n03325584:feather boa, boa 554:n03337140:file, file cabinet, filing cabinet 555:n03344393:fireboat 556:n03345487:fire engine, fire truck 557:n03347037:fire screen, fireguard 558:n03355925:flagpole, flagstaff 559:n03372029:flute, transverse flute 560:n03376595:folding chair 561:n03379051:football helmet 562:n03384352:forklift 563:n03388043:fountain 564:n03388183:fountain pen 565:n03388549:four-poster 566:n03393912:freight car 567:n03394916:French horn, horn 568:n03400231:frying pan, frypan, skillet 569:n03404251:fur coat 570:n03417042:garbage truck, dustcart 571:n03424325:gasmask, respirator, gas helmet 572:n03425413:gas pump, gasoline pump, petrol pump, island dispenser 573:n03443371:goblet 574:n03444034:go-kart 575:n03445777:golf ball 576:n03445924:golfcart, golf cart 577:n03447447:gondola 578:n03447721:gong, tam-tam 579:n03450230:gown 580:n03452741:grand piano, grand 581:n03457902:greenhouse, nursery, glasshouse 582:n03459775:grille, radiator grille 583:n03461385:grocery store, grocery, food market, market 584:n03467068:guillotine 585:n03476684:hair slide 586:n03476991:hair spray 587:n03478589:half track 588:n03481172:hammer 589:n03482405:hamper 590:n03483316:hand blower, blow dryer, blow drier, hair dryer, hair drier 591:n03485407:hand-held computer, hand-held microcomputer 592:n03485794:handkerchief, hankie, hanky, hankey 593:n03492542:hard disc, hard disk, fixed disk 594:n03494278:harmonica, mouth organ, harp, mouth harp 595:n03495258:harp 596:n03496892:harvester, reaper 597:n03498962:hatchet 598:n03527444:holster 599:n03529860:home theater, home theatre 600:n03530642:honeycomb 601:n03532672:hook, claw 602:n03534580:hoopskirt, crinoline 603:n03535780:horizontal bar, high bar 604:n03538406:horse cart, horse-cart 605:n03544143:hourglass 606:n03584254:iPod 607:n03584829:iron, smoothing iron 608:n03590841:jack-o'-lantern 609:n03594734:jean, blue jean, denim 610:n03594945:jeep, landrover 611:n03595614:jersey, T-shirt, tee shirt 612:n03598930:jigsaw puzzle 613:n03599486:jinrikisha, ricksha, rickshaw 614:n03602883:joystick 615:n03617480:kimono 616:n03623198:knee pad 617:n03627232:knot 618:n03630383:lab coat, laboratory coat 619:n03633091:ladle 620:n03637318:lampshade, lamp shade 621:n03642806:laptop, laptop computer 622:n03649909:lawn mower, mower 623:n03657121:lens cap, lens cover 624:n03658185:letter opener, paper knife, paperknife 625:n03661043:library 626:n03662601:lifeboat 627:n03666591:lighter, light, igniter, ignitor 628:n03670208:limousine, limo 629:n03673027:liner, ocean liner 630:n03676483:lipstick, lip rouge 631:n03680355:Loafer 632:n03690938:lotion 633:n03691459:loudspeaker, speaker, speaker unit, loudspeaker system, speaker system 634:n03692522:loupe, jeweler's loupe 635:n03697007:lumbermill, sawmill 636:n03706229:magnetic compass 637:n03709823:mailbag, postbag 638:n03710193:mailbox, letter box 639:n03710637:maillot 640:n03710721:maillot, tank suit 641:n03717622:manhole cover 642:n03720891:maraca 643:n03721384:marimba, xylophone 644:n03724870:mask 645:n03729826:matchstick 646:n03733131:maypole 647:n03733281:maze, labyrinth 648:n03733805:measuring cup 649:n03742115:medicine chest, medicine cabinet 650:n03743016:megalith, megalithic structure 651:n03759954:microphone, mike 652:n03761084:microwave, microwave oven 653:n03763968:military uniform 654:n03764736:milk can 655:n03769881:minibus 656:n03770439:miniskirt, mini 657:n03770679:minivan 658:n03773504:missile 659:n03775071:mitten 660:n03775546:mixing bowl 661:n03776460:mobile home, manufactured home 662:n03777568:Model T 663:n03777754:modem 664:n03781244:monastery 665:n03782006:monitor 666:n03785016:moped 667:n03786901:mortar 668:n03787032:mortarboard 669:n03788195:mosque 670:n03788365:mosquito net 671:n03791053:motor scooter, scooter 672:n03792782:mountain bike, all-terrain bike, off-roader 673:n03792972:mountain tent 674:n03793489:mouse, computer mouse 675:n03794056:mousetrap 676:n03796401:moving van 677:n03803284:muzzle 678:n03804744:nail 679:n03814639:neck brace 680:n03814906:necklace 681:n03825788:nipple 682:n03832673:notebook, notebook computer 683:n03837869:obelisk 684:n03838899:oboe, hautboy, hautbois 685:n03840681:ocarina, sweet potato 686:n03841143:odometer, hodometer, mileometer, milometer 687:n03843555:oil filter 688:n03854065:organ, pipe organ 689:n03857828:oscilloscope, scope, cathode-ray oscilloscope, CRO 690:n03866082:overskirt 691:n03868242:oxcart 692:n03868863:oxygen mask 693:n03871628:packet 694:n03873416:paddle, boat paddle 695:n03874293:paddlewheel, paddle wheel 696:n03874599:padlock 697:n03876231:paintbrush 698:n03877472:pajama, pyjama, pj's, jammies 699:n03877845:palace 700:n03884397:panpipe, pandean pipe, syrinx 701:n03887697:paper towel 702:n03888257:parachute, chute 703:n03888605:parallel bars, bars 704:n03891251:park bench 705:n03891332:parking meter 706:n03895866:passenger car, coach, carriage 707:n03899768:patio, terrace 708:n03902125:pay-phone, pay-station 709:n03903868:pedestal, plinth, footstall 710:n03908618:pencil box, pencil case 711:n03908714:pencil sharpener 712:n03916031:perfume, essence 713:n03920288:Petri dish 714:n03924679:photocopier 715:n03929660:pick, plectrum, plectron 716:n03929855:pickelhaube 717:n03930313:picket fence, paling 718:n03930630:pickup, pickup truck 719:n03933933:pier 720:n03935335:piggy bank, penny bank 721:n03937543:pill bottle 722:n03938244:pillow 723:n03942813:ping-pong ball 724:n03944341:pinwheel 725:n03947888:pirate, pirate ship 726:n03950228:pitcher, ewer 727:n03954731:plane, carpenter's plane, woodworking plane 728:n03956157:planetarium 729:n03958227:plastic bag 730:n03961711:plate rack 731:n03967562:plow, plough 732:n03970156:plunger, plumber's helper 733:n03976467:Polaroid camera, Polaroid Land camera 734:n03976657:pole 735:n03977966:police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria 736:n03980874:poncho 737:n03982430:pool table, billiard table, snooker table 738:n03983396:pop bottle, soda bottle 739:n03991062:pot, flowerpot 740:n03992509:potter's wheel 741:n03995372:power drill 742:n03998194:prayer rug, prayer mat 743:n04004767:printer 744:n04005630:prison, prison house 745:n04008634:projectile, missile 746:n04009552:projector 747:n04019541:puck, hockey puck 748:n04023962:punching bag, punch bag, punching ball, punchball 749:n04026417:purse 750:n04033901:quill, quill pen 751:n04033995:quilt, comforter, comfort, puff 752:n04037443:racer, race car, racing car 753:n04039381:racket, racquet 754:n04040759:radiator 755:n04041544:radio, wireless 756:n04044716:radio telescope, radio reflector 757:n04049303:rain barrel 758:n04065272:recreational vehicle, RV, R.V. 759:n04067472:reel 760:n04069434:reflex camera 761:n04070727:refrigerator, icebox 762:n04074963:remote control, remote 763:n04081281:restaurant, eating house, eating place, eatery 764:n04086273:revolver, six-gun, six-shooter 765:n04090263:rifle 766:n04099969:rocking chair, rocker 767:n04111531:rotisserie 768:n04116512:rubber eraser, rubber, pencil eraser 769:n04118538:rugby ball 770:n04118776:rule, ruler 771:n04120489:running shoe 772:n04125021:safe 773:n04127249:safety pin 774:n04131690:saltshaker, salt shaker 775:n04133789:sandal 776:n04136333:sarong 777:n04141076:sax, saxophone 778:n04141327:scabbard 779:n04141975:scale, weighing machine 780:n04146614:school bus 781:n04147183:schooner 782:n04149813:scoreboard 783:n04152593:screen, CRT screen 784:n04153751:screw 785:n04154565:screwdriver 786:n04162706:seat belt, seatbelt 787:n04179913:sewing machine 788:n04192698:shield, buckler 789:n04200800:shoe shop, shoe-shop, shoe store 790:n04201297:shoji 791:n04204238:shopping basket 792:n04204347:shopping cart 793:n04208210:shovel 794:n04209133:shower cap 795:n04209239:shower curtain 796:n04228054:ski 797:n04229816:ski mask 798:n04235860:sleeping bag 799:n04238763:slide rule, slipstick 800:n04239074:sliding door 801:n04243546:slot, one-armed bandit 802:n04251144:snorkel 803:n04252077:snowmobile 804:n04252225:snowplow, snowplough 805:n04254120:soap dispenser 806:n04254680:soccer ball 807:n04254777:sock 808:n04258138:solar dish, solar collector, solar furnace 809:n04259630:sombrero 810:n04263257:soup bowl 811:n04264628:space bar 812:n04265275:space heater 813:n04266014:space shuttle 814:n04270147:spatula 815:n04273569:speedboat 816:n04275548:spider web, spider's web 817:n04277352:spindle 818:n04285008:sports car, sport car 819:n04286575:spotlight, spot 820:n04296562:stage 821:n04310018:steam locomotive 822:n04311004:steel arch bridge 823:n04311174:steel drum 824:n04317175:stethoscope 825:n04325704:stole 826:n04326547:stone wall 827:n04328186:stopwatch, stop watch 828:n04330267:stove 829:n04332243:strainer 830:n04335435:streetcar, tram, tramcar, trolley, trolley car 831:n04336792:stretcher 832:n04344873:studio couch, day bed 833:n04346328:stupa, tope 834:n04347754:submarine, pigboat, sub, U-boat 835:n04350905:suit, suit of clothes 836:n04355338:sundial 837:n04355933:sunglass 838:n04356056:sunglasses, dark glasses, shades 839:n04357314:sunscreen, sunblock, sun blocker 840:n04366367:suspension bridge 841:n04367480:swab, swob, mop 842:n04370456:sweatshirt 843:n04371430:swimming trunks, bathing trunks 844:n04371774:swing 845:n04372370:switch, electric switch, electrical switch 846:n04376876:syringe 847:n04380533:table lamp 848:n04389033:tank, army tank, armored combat vehicle, armoured combat vehicle 849:n04392985:tape player 850:n04398044:teapot 851:n04399382:teddy, teddy bear 852:n04404412:television, television system 853:n04409515:tennis ball 854:n04417672:thatch, thatched roof 855:n04418357:theater curtain, theatre curtain 856:n04423845:thimble 857:n04428191:thresher, thrasher, threshing machine 858:n04429376:throne 859:n04435653:tile roof 860:n04442312:toaster 861:n04443257:tobacco shop, tobacconist shop, tobacconist 862:n04447861:toilet seat 863:n04456115:torch 864:n04458633:totem pole 865:n04461696:tow truck, tow car, wrecker 866:n04462240:toyshop 867:n04465501:tractor 868:n04467665:trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi 869:n04476259:tray 870:n04479046:trench coat 871:n04482393:tricycle, trike, velocipede 872:n04483307:trimaran 873:n04485082:tripod 874:n04486054:triumphal arch 875:n04487081:trolleybus, trolley coach, trackless trolley 876:n04487394:trombone 877:n04493381:tub, vat 878:n04501370:turnstile 879:n04505470:typewriter keyboard 880:n04507155:umbrella 881:n04509417:unicycle, monocycle 882:n04515003:upright, upright piano 883:n04517823:vacuum, vacuum cleaner 884:n04522168:vase 885:n04523525:vault 886:n04525038:velvet 887:n04525305:vending machine 888:n04532106:vestment 889:n04532670:viaduct 890:n04536866:violin, fiddle 891:n04540053:volleyball 892:n04542943:waffle iron 893:n04548280:wall clock 894:n04548362:wallet, billfold, notecase, pocketbook 895:n04550184:wardrobe, closet, press 896:n04552348:warplane, military plane 897:n04553703:washbasin, handbasin, washbowl, lavabo, wash-hand basin 898:n04554684:washer, automatic washer, washing machine 899:n04557648:water bottle 900:n04560804:water jug 901:n04562935:water tower 902:n04579145:whiskey jug 903:n04579432:whistle 904:n04584207:wig 905:n04589890:window screen 906:n04590129:window shade 907:n04591157:Windsor tie 908:n04591713:wine bottle 909:n04592741:wing 910:n04596742:wok 911:n04597913:wooden spoon 912:n04599235:wool, woolen, woollen 913:n04604644:worm fence, snake fence, snake-rail fence, Virginia fence 914:n04606251:wreck 915:n04612504:yawl 916:n04613696:yurt 917:n06359193:web site, website, internet site, site 918:n06596364:comic book 919:n06785654:crossword puzzle, crossword 920:n06794110:street sign 921:n06874185:traffic light, traffic signal, stoplight 922:n07248320:book jacket, dust cover, dust jacket, dust wrapper 923:n07565083:menu 924:n07579787:plate 925:n07583066:guacamole 926:n07584110:consomme 927:n07590611:hot pot, hotpot 928:n07613480:trifle 929:n07614500:ice cream, icecream 930:n07615774:ice lolly, lolly, lollipop, popsicle 931:n07684084:French loaf 932:n07693725:bagel, beigel 933:n07695742:pretzel 934:n07697313:cheeseburger 935:n07697537:hotdog, hot dog, red hot 936:n07711569:mashed potato 937:n07714571:head cabbage 938:n07714990:broccoli 939:n07715103:cauliflower 940:n07716358:zucchini, courgette 941:n07716906:spaghetti squash 942:n07717410:acorn squash 943:n07717556:butternut squash 944:n07718472:cucumber, cuke 945:n07718747:artichoke, globe artichoke 946:n07720875:bell pepper 947:n07730033:cardoon 948:n07734744:mushroom 949:n07742313:Granny Smith 950:n07745940:strawberry 951:n07747607:orange 952:n07749582:lemon 953:n07753113:fig 954:n07753275:pineapple, ananas 955:n07753592:banana 956:n07754684:jackfruit, jak, jack 957:n07760859:custard apple 958:n07768694:pomegranate 959:n07802026:hay 960:n07831146:carbonara 961:n07836838:chocolate sauce, chocolate syrup 962:n07860988:dough 963:n07871810:meat loaf, meatloaf 964:n07873807:pizza, pizza pie 965:n07875152:potpie 966:n07880968:burrito 967:n07892512:red wine 968:n07920052:espresso 969:n07930864:cup 970:n07932039:eggnog 971:n09193705:alp 972:n09229709:bubble 973:n09246464:cliff, drop, drop-off 974:n09256479:coral reef 975:n09288635:geyser 976:n09332890:lakeside, lakeshore 977:n09399592:promontory, headland, head, foreland 978:n09421951:sandbar, sand bar 979:n09428293:seashore, coast, seacoast, sea-coast 980:n09468604:valley, vale 981:n09472597:volcano 982:n09835506:ballplayer, baseball player 983:n10148035:groom, bridegroom 984:n10565667:scuba diver 985:n11879895:rapeseed 986:n11939491:daisy 987:n12057211:yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum 988:n12144580:corn 989:n12267677:acorn 990:n12620546:hip, rose hip, rosehip 991:n12768682:buckeye, horse chestnut, conker 992:n12985857:coral fungus 993:n12998815:agaric 994:n13037406:gyromitra 995:n13040303:stinkhorn, carrion fungus 996:n13044778:earthstar 997:n13052670:hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa 998:n13054560:bolete 999:n13133613:ear, spike, capitulum 1000:n15075141:toilet tissue, toilet paper, bathroom tissue ================================================ FILE: labels/labels.py ================================================ #!/usr/bin/env python3 new_label = "labels.txt" imagenet_map = "synset_words.txt" label2num = "label2num.txt" num_dict = {} out_file = open(label2num, 'w') with open(new_label, 'r') as f: for line in f: if line: num = int(line.split(':')[0]) lab = line.split(':')[1].strip() num_dict[lab] = num print(len(num_dict)) with open(imagenet_map, 'r') as f: for line in f: if line: nn = line[:9] lab = line[10:].strip() print(nn, lab) num = num_dict[lab] out_file.write("{}:{}:{}\n".format(num, nn, lab)) ================================================ FILE: labels/labels.txt ================================================ 0:background 1:tench, Tinca tinca 2:goldfish, Carassius auratus 3:great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias 4:tiger shark, Galeocerdo cuvieri 5:hammerhead, hammerhead shark 6:electric ray, crampfish, numbfish, torpedo 7:stingray 8:cock 9:hen 10:ostrich, Struthio camelus 11:brambling, Fringilla montifringilla 12:goldfinch, Carduelis carduelis 13:house finch, linnet, Carpodacus mexicanus 14:junco, snowbird 15:indigo bunting, indigo finch, indigo bird, Passerina cyanea 16:robin, American robin, Turdus migratorius 17:bulbul 18:jay 19:magpie 20:chickadee 21:water ouzel, dipper 22:kite 23:bald eagle, American eagle, Haliaeetus leucocephalus 24:vulture 25:great grey owl, great gray owl, Strix nebulosa 26:European fire salamander, Salamandra salamandra 27:common newt, Triturus vulgaris 28:eft 29:spotted salamander, Ambystoma maculatum 30:axolotl, mud puppy, Ambystoma mexicanum 31:bullfrog, Rana catesbeiana 32:tree frog, tree-frog 33:tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui 34:loggerhead, loggerhead turtle, Caretta caretta 35:leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea 36:mud turtle 37:terrapin 38:box turtle, box tortoise 39:banded gecko 40:common iguana, iguana, Iguana iguana 41:American chameleon, anole, Anolis carolinensis 42:whiptail, whiptail lizard 43:agama 44:frilled lizard, Chlamydosaurus kingi 45:alligator lizard 46:Gila monster, Heloderma suspectum 47:green lizard, Lacerta viridis 48:African chameleon, Chamaeleo chamaeleon 49:Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis 50:African crocodile, Nile crocodile, Crocodylus niloticus 51:American alligator, Alligator mississipiensis 52:triceratops 53:thunder snake, worm snake, Carphophis amoenus 54:ringneck snake, ring-necked snake, ring snake 55:hognose snake, puff adder, sand viper 56:green snake, grass snake 57:king snake, kingsnake 58:garter snake, grass snake 59:water snake 60:vine snake 61:night snake, Hypsiglena torquata 62:boa constrictor, Constrictor constrictor 63:rock python, rock snake, Python sebae 64:Indian cobra, Naja naja 65:green mamba 66:sea snake 67:horned viper, cerastes, sand viper, horned asp, Cerastes cornutus 68:diamondback, diamondback rattlesnake, Crotalus adamanteus 69:sidewinder, horned rattlesnake, Crotalus cerastes 70:trilobite 71:harvestman, daddy longlegs, Phalangium opilio 72:scorpion 73:black and gold garden spider, Argiope aurantia 74:barn spider, Araneus cavaticus 75:garden spider, Aranea diademata 76:black widow, Latrodectus mactans 77:tarantula 78:wolf spider, hunting spider 79:tick 80:centipede 81:black grouse 82:ptarmigan 83:ruffed grouse, partridge, Bonasa umbellus 84:prairie chicken, prairie grouse, prairie fowl 85:peacock 86:quail 87:partridge 88:African grey, African gray, Psittacus erithacus 89:macaw 90:sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita 91:lorikeet 92:coucal 93:bee eater 94:hornbill 95:hummingbird 96:jacamar 97:toucan 98:drake 99:red-breasted merganser, Mergus serrator 100:goose 101:black swan, Cygnus atratus 102:tusker 103:echidna, spiny anteater, anteater 104:platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus 105:wallaby, brush kangaroo 106:koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus 107:wombat 108:jellyfish 109:sea anemone, anemone 110:brain coral 111:flatworm, platyhelminth 112:nematode, nematode worm, roundworm 113:conch 114:snail 115:slug 116:sea slug, nudibranch 117:chiton, coat-of-mail shell, sea cradle, polyplacophore 118:chambered nautilus, pearly nautilus, nautilus 119:Dungeness crab, Cancer magister 120:rock crab, Cancer irroratus 121:fiddler crab 122:king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica 123:American lobster, Northern lobster, Maine lobster, Homarus americanus 124:spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish 125:crayfish, crawfish, crawdad, crawdaddy 126:hermit crab 127:isopod 128:white stork, Ciconia ciconia 129:black stork, Ciconia nigra 130:spoonbill 131:flamingo 132:little blue heron, Egretta caerulea 133:American egret, great white heron, Egretta albus 134:bittern 135:crane 136:limpkin, Aramus pictus 137:European gallinule, Porphyrio porphyrio 138:American coot, marsh hen, mud hen, water hen, Fulica americana 139:bustard 140:ruddy turnstone, Arenaria interpres 141:red-backed sandpiper, dunlin, Erolia alpina 142:redshank, Tringa totanus 143:dowitcher 144:oystercatcher, oyster catcher 145:pelican 146:king penguin, Aptenodytes patagonica 147:albatross, mollymawk 148:grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus 149:killer whale, killer, orca, grampus, sea wolf, Orcinus orca 150:dugong, Dugong dugon 151:sea lion 152:Chihuahua 153:Japanese spaniel 154:Maltese dog, Maltese terrier, Maltese 155:Pekinese, Pekingese, Peke 156:Shih-Tzu 157:Blenheim spaniel 158:papillon 159:toy terrier 160:Rhodesian ridgeback 161:Afghan hound, Afghan 162:basset, basset hound 163:beagle 164:bloodhound, sleuthhound 165:bluetick 166:black-and-tan coonhound 167:Walker hound, Walker foxhound 168:English foxhound 169:redbone 170:borzoi, Russian wolfhound 171:Irish wolfhound 172:Italian greyhound 173:whippet 174:Ibizan hound, Ibizan Podenco 175:Norwegian elkhound, elkhound 176:otterhound, otter hound 177:Saluki, gazelle hound 178:Scottish deerhound, deerhound 179:Weimaraner 180:Staffordshire bullterrier, Staffordshire bull terrier 181:American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier 182:Bedlington terrier 183:Border terrier 184:Kerry blue terrier 185:Irish terrier 186:Norfolk terrier 187:Norwich terrier 188:Yorkshire terrier 189:wire-haired fox terrier 190:Lakeland terrier 191:Sealyham terrier, Sealyham 192:Airedale, Airedale terrier 193:cairn, cairn terrier 194:Australian terrier 195:Dandie Dinmont, Dandie Dinmont terrier 196:Boston bull, Boston terrier 197:miniature schnauzer 198:giant schnauzer 199:standard schnauzer 200:Scotch terrier, Scottish terrier, Scottie 201:Tibetan terrier, chrysanthemum dog 202:silky terrier, Sydney silky 203:soft-coated wheaten terrier 204:West Highland white terrier 205:Lhasa, Lhasa apso 206:flat-coated retriever 207:curly-coated retriever 208:golden retriever 209:Labrador retriever 210:Chesapeake Bay retriever 211:German short-haired pointer 212:vizsla, Hungarian pointer 213:English setter 214:Irish setter, red setter 215:Gordon setter 216:Brittany spaniel 217:clumber, clumber spaniel 218:English springer, English springer spaniel 219:Welsh springer spaniel 220:cocker spaniel, English cocker spaniel, cocker 221:Sussex spaniel 222:Irish water spaniel 223:kuvasz 224:schipperke 225:groenendael 226:malinois 227:briard 228:kelpie 229:komondor 230:Old English sheepdog, bobtail 231:Shetland sheepdog, Shetland sheep dog, Shetland 232:collie 233:Border collie 234:Bouvier des Flandres, Bouviers des Flandres 235:Rottweiler 236:German shepherd, German shepherd dog, German police dog, alsatian 237:Doberman, Doberman pinscher 238:miniature pinscher 239:Greater Swiss Mountain dog 240:Bernese mountain dog 241:Appenzeller 242:EntleBucher 243:boxer 244:bull mastiff 245:Tibetan mastiff 246:French bulldog 247:Great Dane 248:Saint Bernard, St Bernard 249:Eskimo dog, husky 250:malamute, malemute, Alaskan malamute 251:Siberian husky 252:dalmatian, coach dog, carriage dog 253:affenpinscher, monkey pinscher, monkey dog 254:basenji 255:pug, pug-dog 256:Leonberg 257:Newfoundland, Newfoundland dog 258:Great Pyrenees 259:Samoyed, Samoyede 260:Pomeranian 261:chow, chow chow 262:keeshond 263:Brabancon griffon 264:Pembroke, Pembroke Welsh corgi 265:Cardigan, Cardigan Welsh corgi 266:toy poodle 267:miniature poodle 268:standard poodle 269:Mexican hairless 270:timber wolf, grey wolf, gray wolf, Canis lupus 271:white wolf, Arctic wolf, Canis lupus tundrarum 272:red wolf, maned wolf, Canis rufus, Canis niger 273:coyote, prairie wolf, brush wolf, Canis latrans 274:dingo, warrigal, warragal, Canis dingo 275:dhole, Cuon alpinus 276:African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus 277:hyena, hyaena 278:red fox, Vulpes vulpes 279:kit fox, Vulpes macrotis 280:Arctic fox, white fox, Alopex lagopus 281:grey fox, gray fox, Urocyon cinereoargenteus 282:tabby, tabby cat 283:tiger cat 284:Persian cat 285:Siamese cat, Siamese 286:Egyptian cat 287:cougar, puma, catamount, mountain lion, painter, panther, Felis concolor 288:lynx, catamount 289:leopard, Panthera pardus 290:snow leopard, ounce, Panthera uncia 291:jaguar, panther, Panthera onca, Felis onca 292:lion, king of beasts, Panthera leo 293:tiger, Panthera tigris 294:cheetah, chetah, Acinonyx jubatus 295:brown bear, bruin, Ursus arctos 296:American black bear, black bear, Ursus americanus, Euarctos americanus 297:ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus 298:sloth bear, Melursus ursinus, Ursus ursinus 299:mongoose 300:meerkat, mierkat 301:tiger beetle 302:ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle 303:ground beetle, carabid beetle 304:long-horned beetle, longicorn, longicorn beetle 305:leaf beetle, chrysomelid 306:dung beetle 307:rhinoceros beetle 308:weevil 309:fly 310:bee 311:ant, emmet, pismire 312:grasshopper, hopper 313:cricket 314:walking stick, walkingstick, stick insect 315:cockroach, roach 316:mantis, mantid 317:cicada, cicala 318:leafhopper 319:lacewing, lacewing fly 320:dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk 321:damselfly 322:admiral 323:ringlet, ringlet butterfly 324:monarch, monarch butterfly, milkweed butterfly, Danaus plexippus 325:cabbage butterfly 326:sulphur butterfly, sulfur butterfly 327:lycaenid, lycaenid butterfly 328:starfish, sea star 329:sea urchin 330:sea cucumber, holothurian 331:wood rabbit, cottontail, cottontail rabbit 332:hare 333:Angora, Angora rabbit 334:hamster 335:porcupine, hedgehog 336:fox squirrel, eastern fox squirrel, Sciurus niger 337:marmot 338:beaver 339:guinea pig, Cavia cobaya 340:sorrel 341:zebra 342:hog, pig, grunter, squealer, Sus scrofa 343:wild boar, boar, Sus scrofa 344:warthog 345:hippopotamus, hippo, river horse, Hippopotamus amphibius 346:ox 347:water buffalo, water ox, Asiatic buffalo, Bubalus bubalis 348:bison 349:ram, tup 350:bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis 351:ibex, Capra ibex 352:hartebeest 353:impala, Aepyceros melampus 354:gazelle 355:Arabian camel, dromedary, Camelus dromedarius 356:llama 357:weasel 358:mink 359:polecat, fitch, foulmart, foumart, Mustela putorius 360:black-footed ferret, ferret, Mustela nigripes 361:otter 362:skunk, polecat, wood pussy 363:badger 364:armadillo 365:three-toed sloth, ai, Bradypus tridactylus 366:orangutan, orang, orangutang, Pongo pygmaeus 367:gorilla, Gorilla gorilla 368:chimpanzee, chimp, Pan troglodytes 369:gibbon, Hylobates lar 370:siamang, Hylobates syndactylus, Symphalangus syndactylus 371:guenon, guenon monkey 372:patas, hussar monkey, Erythrocebus patas 373:baboon 374:macaque 375:langur 376:colobus, colobus monkey 377:proboscis monkey, Nasalis larvatus 378:marmoset 379:capuchin, ringtail, Cebus capucinus 380:howler monkey, howler 381:titi, titi monkey 382:spider monkey, Ateles geoffroyi 383:squirrel monkey, Saimiri sciureus 384:Madagascar cat, ring-tailed lemur, Lemur catta 385:indri, indris, Indri indri, Indri brevicaudatus 386:Indian elephant, Elephas maximus 387:African elephant, Loxodonta africana 388:lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens 389:giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca 390:barracouta, snoek 391:eel 392:coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch 393:rock beauty, Holocanthus tricolor 394:anemone fish 395:sturgeon 396:gar, garfish, garpike, billfish, Lepisosteus osseus 397:lionfish 398:puffer, pufferfish, blowfish, globefish 399:abacus 400:abaya 401:academic gown, academic robe, judge's robe 402:accordion, piano accordion, squeeze box 403:acoustic guitar 404:aircraft carrier, carrier, flattop, attack aircraft carrier 405:airliner 406:airship, dirigible 407:altar 408:ambulance 409:amphibian, amphibious vehicle 410:analog clock 411:apiary, bee house 412:apron 413:ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin 414:assault rifle, assault gun 415:backpack, back pack, knapsack, packsack, rucksack, haversack 416:bakery, bakeshop, bakehouse 417:balance beam, beam 418:balloon 419:ballpoint, ballpoint pen, ballpen, Biro 420:Band Aid 421:banjo 422:bannister, banister, balustrade, balusters, handrail 423:barbell 424:barber chair 425:barbershop 426:barn 427:barometer 428:barrel, cask 429:barrow, garden cart, lawn cart, wheelbarrow 430:baseball 431:basketball 432:bassinet 433:bassoon 434:bathing cap, swimming cap 435:bath towel 436:bathtub, bathing tub, bath, tub 437:beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon 438:beacon, lighthouse, beacon light, pharos 439:beaker 440:bearskin, busby, shako 441:beer bottle 442:beer glass 443:bell cote, bell cot 444:bib 445:bicycle-built-for-two, tandem bicycle, tandem 446:bikini, two-piece 447:binder, ring-binder 448:binoculars, field glasses, opera glasses 449:birdhouse 450:boathouse 451:bobsled, bobsleigh, bob 452:bolo tie, bolo, bola tie, bola 453:bonnet, poke bonnet 454:bookcase 455:bookshop, bookstore, bookstall 456:bottlecap 457:bow 458:bow tie, bow-tie, bowtie 459:brass, memorial tablet, plaque 460:brassiere, bra, bandeau 461:breakwater, groin, groyne, mole, bulwark, seawall, jetty 462:breastplate, aegis, egis 463:broom 464:bucket, pail 465:buckle 466:bulletproof vest 467:bullet train, bullet 468:butcher shop, meat market 469:cab, hack, taxi, taxicab 470:caldron, cauldron 471:candle, taper, wax light 472:cannon 473:canoe 474:can opener, tin opener 475:cardigan 476:car mirror 477:carousel, carrousel, merry-go-round, roundabout, whirligig 478:carpenter's kit, tool kit 479:carton 480:car wheel 481:cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM 482:cassette 483:cassette player 484:castle 485:catamaran 486:CD player 487:cello, violoncello 488:cellular telephone, cellular phone, cellphone, cell, mobile phone 489:chain 490:chainlink fence 491:chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour 492:chain saw, chainsaw 493:chest 494:chiffonier, commode 495:chime, bell, gong 496:china cabinet, china closet 497:Christmas stocking 498:church, church building 499:cinema, movie theater, movie theatre, movie house, picture palace 500:cleaver, meat cleaver, chopper 501:cliff dwelling 502:cloak 503:clog, geta, patten, sabot 504:cocktail shaker 505:coffee mug 506:coffeepot 507:coil, spiral, volute, whorl, helix 508:combination lock 509:computer keyboard, keypad 510:confectionery, confectionary, candy store 511:container ship, containership, container vessel 512:convertible 513:corkscrew, bottle screw 514:cornet, horn, trumpet, trump 515:cowboy boot 516:cowboy hat, ten-gallon hat 517:cradle 518:crane 519:crash helmet 520:crate 521:crib, cot 522:Crock Pot 523:croquet ball 524:crutch 525:cuirass 526:dam, dike, dyke 527:desk 528:desktop computer 529:dial telephone, dial phone 530:diaper, nappy, napkin 531:digital clock 532:digital watch 533:dining table, board 534:dishrag, dishcloth 535:dishwasher, dish washer, dishwashing machine 536:disk brake, disc brake 537:dock, dockage, docking facility 538:dogsled, dog sled, dog sleigh 539:dome 540:doormat, welcome mat 541:drilling platform, offshore rig 542:drum, membranophone, tympan 543:drumstick 544:dumbbell 545:Dutch oven 546:electric fan, blower 547:electric guitar 548:electric locomotive 549:entertainment center 550:envelope 551:espresso maker 552:face powder 553:feather boa, boa 554:file, file cabinet, filing cabinet 555:fireboat 556:fire engine, fire truck 557:fire screen, fireguard 558:flagpole, flagstaff 559:flute, transverse flute 560:folding chair 561:football helmet 562:forklift 563:fountain 564:fountain pen 565:four-poster 566:freight car 567:French horn, horn 568:frying pan, frypan, skillet 569:fur coat 570:garbage truck, dustcart 571:gasmask, respirator, gas helmet 572:gas pump, gasoline pump, petrol pump, island dispenser 573:goblet 574:go-kart 575:golf ball 576:golfcart, golf cart 577:gondola 578:gong, tam-tam 579:gown 580:grand piano, grand 581:greenhouse, nursery, glasshouse 582:grille, radiator grille 583:grocery store, grocery, food market, market 584:guillotine 585:hair slide 586:hair spray 587:half track 588:hammer 589:hamper 590:hand blower, blow dryer, blow drier, hair dryer, hair drier 591:hand-held computer, hand-held microcomputer 592:handkerchief, hankie, hanky, hankey 593:hard disc, hard disk, fixed disk 594:harmonica, mouth organ, harp, mouth harp 595:harp 596:harvester, reaper 597:hatchet 598:holster 599:home theater, home theatre 600:honeycomb 601:hook, claw 602:hoopskirt, crinoline 603:horizontal bar, high bar 604:horse cart, horse-cart 605:hourglass 606:iPod 607:iron, smoothing iron 608:jack-o'-lantern 609:jean, blue jean, denim 610:jeep, landrover 611:jersey, T-shirt, tee shirt 612:jigsaw puzzle 613:jinrikisha, ricksha, rickshaw 614:joystick 615:kimono 616:knee pad 617:knot 618:lab coat, laboratory coat 619:ladle 620:lampshade, lamp shade 621:laptop, laptop computer 622:lawn mower, mower 623:lens cap, lens cover 624:letter opener, paper knife, paperknife 625:library 626:lifeboat 627:lighter, light, igniter, ignitor 628:limousine, limo 629:liner, ocean liner 630:lipstick, lip rouge 631:Loafer 632:lotion 633:loudspeaker, speaker, speaker unit, loudspeaker system, speaker system 634:loupe, jeweler's loupe 635:lumbermill, sawmill 636:magnetic compass 637:mailbag, postbag 638:mailbox, letter box 639:maillot 640:maillot, tank suit 641:manhole cover 642:maraca 643:marimba, xylophone 644:mask 645:matchstick 646:maypole 647:maze, labyrinth 648:measuring cup 649:medicine chest, medicine cabinet 650:megalith, megalithic structure 651:microphone, mike 652:microwave, microwave oven 653:military uniform 654:milk can 655:minibus 656:miniskirt, mini 657:minivan 658:missile 659:mitten 660:mixing bowl 661:mobile home, manufactured home 662:Model T 663:modem 664:monastery 665:monitor 666:moped 667:mortar 668:mortarboard 669:mosque 670:mosquito net 671:motor scooter, scooter 672:mountain bike, all-terrain bike, off-roader 673:mountain tent 674:mouse, computer mouse 675:mousetrap 676:moving van 677:muzzle 678:nail 679:neck brace 680:necklace 681:nipple 682:notebook, notebook computer 683:obelisk 684:oboe, hautboy, hautbois 685:ocarina, sweet potato 686:odometer, hodometer, mileometer, milometer 687:oil filter 688:organ, pipe organ 689:oscilloscope, scope, cathode-ray oscilloscope, CRO 690:overskirt 691:oxcart 692:oxygen mask 693:packet 694:paddle, boat paddle 695:paddlewheel, paddle wheel 696:padlock 697:paintbrush 698:pajama, pyjama, pj's, jammies 699:palace 700:panpipe, pandean pipe, syrinx 701:paper towel 702:parachute, chute 703:parallel bars, bars 704:park bench 705:parking meter 706:passenger car, coach, carriage 707:patio, terrace 708:pay-phone, pay-station 709:pedestal, plinth, footstall 710:pencil box, pencil case 711:pencil sharpener 712:perfume, essence 713:Petri dish 714:photocopier 715:pick, plectrum, plectron 716:pickelhaube 717:picket fence, paling 718:pickup, pickup truck 719:pier 720:piggy bank, penny bank 721:pill bottle 722:pillow 723:ping-pong ball 724:pinwheel 725:pirate, pirate ship 726:pitcher, ewer 727:plane, carpenter's plane, woodworking plane 728:planetarium 729:plastic bag 730:plate rack 731:plow, plough 732:plunger, plumber's helper 733:Polaroid camera, Polaroid Land camera 734:pole 735:police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria 736:poncho 737:pool table, billiard table, snooker table 738:pop bottle, soda bottle 739:pot, flowerpot 740:potter's wheel 741:power drill 742:prayer rug, prayer mat 743:printer 744:prison, prison house 745:projectile, missile 746:projector 747:puck, hockey puck 748:punching bag, punch bag, punching ball, punchball 749:purse 750:quill, quill pen 751:quilt, comforter, comfort, puff 752:racer, race car, racing car 753:racket, racquet 754:radiator 755:radio, wireless 756:radio telescope, radio reflector 757:rain barrel 758:recreational vehicle, RV, R.V. 759:reel 760:reflex camera 761:refrigerator, icebox 762:remote control, remote 763:restaurant, eating house, eating place, eatery 764:revolver, six-gun, six-shooter 765:rifle 766:rocking chair, rocker 767:rotisserie 768:rubber eraser, rubber, pencil eraser 769:rugby ball 770:rule, ruler 771:running shoe 772:safe 773:safety pin 774:saltshaker, salt shaker 775:sandal 776:sarong 777:sax, saxophone 778:scabbard 779:scale, weighing machine 780:school bus 781:schooner 782:scoreboard 783:screen, CRT screen 784:screw 785:screwdriver 786:seat belt, seatbelt 787:sewing machine 788:shield, buckler 789:shoe shop, shoe-shop, shoe store 790:shoji 791:shopping basket 792:shopping cart 793:shovel 794:shower cap 795:shower curtain 796:ski 797:ski mask 798:sleeping bag 799:slide rule, slipstick 800:sliding door 801:slot, one-armed bandit 802:snorkel 803:snowmobile 804:snowplow, snowplough 805:soap dispenser 806:soccer ball 807:sock 808:solar dish, solar collector, solar furnace 809:sombrero 810:soup bowl 811:space bar 812:space heater 813:space shuttle 814:spatula 815:speedboat 816:spider web, spider's web 817:spindle 818:sports car, sport car 819:spotlight, spot 820:stage 821:steam locomotive 822:steel arch bridge 823:steel drum 824:stethoscope 825:stole 826:stone wall 827:stopwatch, stop watch 828:stove 829:strainer 830:streetcar, tram, tramcar, trolley, trolley car 831:stretcher 832:studio couch, day bed 833:stupa, tope 834:submarine, pigboat, sub, U-boat 835:suit, suit of clothes 836:sundial 837:sunglass 838:sunglasses, dark glasses, shades 839:sunscreen, sunblock, sun blocker 840:suspension bridge 841:swab, swob, mop 842:sweatshirt 843:swimming trunks, bathing trunks 844:swing 845:switch, electric switch, electrical switch 846:syringe 847:table lamp 848:tank, army tank, armored combat vehicle, armoured combat vehicle 849:tape player 850:teapot 851:teddy, teddy bear 852:television, television system 853:tennis ball 854:thatch, thatched roof 855:theater curtain, theatre curtain 856:thimble 857:thresher, thrasher, threshing machine 858:throne 859:tile roof 860:toaster 861:tobacco shop, tobacconist shop, tobacconist 862:toilet seat 863:torch 864:totem pole 865:tow truck, tow car, wrecker 866:toyshop 867:tractor 868:trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi 869:tray 870:trench coat 871:tricycle, trike, velocipede 872:trimaran 873:tripod 874:triumphal arch 875:trolleybus, trolley coach, trackless trolley 876:trombone 877:tub, vat 878:turnstile 879:typewriter keyboard 880:umbrella 881:unicycle, monocycle 882:upright, upright piano 883:vacuum, vacuum cleaner 884:vase 885:vault 886:velvet 887:vending machine 888:vestment 889:viaduct 890:violin, fiddle 891:volleyball 892:waffle iron 893:wall clock 894:wallet, billfold, notecase, pocketbook 895:wardrobe, closet, press 896:warplane, military plane 897:washbasin, handbasin, washbowl, lavabo, wash-hand basin 898:washer, automatic washer, washing machine 899:water bottle 900:water jug 901:water tower 902:whiskey jug 903:whistle 904:wig 905:window screen 906:window shade 907:Windsor tie 908:wine bottle 909:wing 910:wok 911:wooden spoon 912:wool, woolen, woollen 913:worm fence, snake fence, snake-rail fence, Virginia fence 914:wreck 915:yawl 916:yurt 917:web site, website, internet site, site 918:comic book 919:crossword puzzle, crossword 920:street sign 921:traffic light, traffic signal, stoplight 922:book jacket, dust cover, dust jacket, dust wrapper 923:menu 924:plate 925:guacamole 926:consomme 927:hot pot, hotpot 928:trifle 929:ice cream, icecream 930:ice lolly, lolly, lollipop, popsicle 931:French loaf 932:bagel, beigel 933:pretzel 934:cheeseburger 935:hotdog, hot dog, red hot 936:mashed potato 937:head cabbage 938:broccoli 939:cauliflower 940:zucchini, courgette 941:spaghetti squash 942:acorn squash 943:butternut squash 944:cucumber, cuke 945:artichoke, globe artichoke 946:bell pepper 947:cardoon 948:mushroom 949:Granny Smith 950:strawberry 951:orange 952:lemon 953:fig 954:pineapple, ananas 955:banana 956:jackfruit, jak, jack 957:custard apple 958:pomegranate 959:hay 960:carbonara 961:chocolate sauce, chocolate syrup 962:dough 963:meat loaf, meatloaf 964:pizza, pizza pie 965:potpie 966:burrito 967:red wine 968:espresso 969:cup 970:eggnog 971:alp 972:bubble 973:cliff, drop, drop-off 974:coral reef 975:geyser 976:lakeside, lakeshore 977:promontory, headland, head, foreland 978:sandbar, sand bar 979:seashore, coast, seacoast, sea-coast 980:valley, vale 981:volcano 982:ballplayer, baseball player 983:groom, bridegroom 984:scuba diver 985:rapeseed 986:daisy 987:yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum 988:corn 989:acorn 990:hip, rose hip, rosehip 991:buckeye, horse chestnut, conker 992:coral fungus 993:agaric 994:gyromitra 995:stinkhorn, carrion fungus 996:earthstar 997:hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa 998:bolete 999:ear, spike, capitulum 1000:toilet tissue, toilet paper, bathroom tissue ================================================ FILE: labels/synset_words.txt ================================================ n01440764 tench, Tinca tinca n01443537 goldfish, Carassius auratus n01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias n01491361 tiger shark, Galeocerdo cuvieri n01494475 hammerhead, hammerhead shark n01496331 electric ray, crampfish, numbfish, torpedo n01498041 stingray n01514668 cock n01514859 hen n01518878 ostrich, Struthio camelus n01530575 brambling, Fringilla montifringilla n01531178 goldfinch, Carduelis carduelis n01532829 house finch, linnet, Carpodacus mexicanus n01534433 junco, snowbird n01537544 indigo bunting, indigo finch, indigo bird, Passerina cyanea n01558993 robin, American robin, Turdus migratorius n01560419 bulbul n01580077 jay n01582220 magpie n01592084 chickadee n01601694 water ouzel, dipper n01608432 kite n01614925 bald eagle, American eagle, Haliaeetus leucocephalus n01616318 vulture n01622779 great grey owl, great gray owl, Strix nebulosa n01629819 European fire salamander, Salamandra salamandra n01630670 common newt, Triturus vulgaris n01631663 eft n01632458 spotted salamander, Ambystoma maculatum n01632777 axolotl, mud puppy, Ambystoma mexicanum n01641577 bullfrog, Rana catesbeiana n01644373 tree frog, tree-frog n01644900 tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui n01664065 loggerhead, loggerhead turtle, Caretta caretta n01665541 leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea n01667114 mud turtle n01667778 terrapin n01669191 box turtle, box tortoise n01675722 banded gecko n01677366 common iguana, iguana, Iguana iguana n01682714 American chameleon, anole, Anolis carolinensis n01685808 whiptail, whiptail lizard n01687978 agama n01688243 frilled lizard, Chlamydosaurus kingi n01689811 alligator lizard n01692333 Gila monster, Heloderma suspectum n01693334 green lizard, Lacerta viridis n01694178 African chameleon, Chamaeleo chamaeleon n01695060 Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis n01697457 African crocodile, Nile crocodile, Crocodylus niloticus n01698640 American alligator, Alligator mississipiensis n01704323 triceratops n01728572 thunder snake, worm snake, Carphophis amoenus n01728920 ringneck snake, ring-necked snake, ring snake n01729322 hognose snake, puff adder, sand viper n01729977 green snake, grass snake n01734418 king snake, kingsnake n01735189 garter snake, grass snake n01737021 water snake n01739381 vine snake n01740131 night snake, Hypsiglena torquata n01742172 boa constrictor, Constrictor constrictor n01744401 rock python, rock snake, Python sebae n01748264 Indian cobra, Naja naja n01749939 green mamba n01751748 sea snake n01753488 horned viper, cerastes, sand viper, horned asp, Cerastes cornutus n01755581 diamondback, diamondback rattlesnake, Crotalus adamanteus n01756291 sidewinder, horned rattlesnake, Crotalus cerastes n01768244 trilobite n01770081 harvestman, daddy longlegs, Phalangium opilio n01770393 scorpion n01773157 black and gold garden spider, Argiope aurantia n01773549 barn spider, Araneus cavaticus n01773797 garden spider, Aranea diademata n01774384 black widow, Latrodectus mactans n01774750 tarantula n01775062 wolf spider, hunting spider n01776313 tick n01784675 centipede n01795545 black grouse n01796340 ptarmigan n01797886 ruffed grouse, partridge, Bonasa umbellus n01798484 prairie chicken, prairie grouse, prairie fowl n01806143 peacock n01806567 quail n01807496 partridge n01817953 African grey, African gray, Psittacus erithacus n01818515 macaw n01819313 sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita n01820546 lorikeet n01824575 coucal n01828970 bee eater n01829413 hornbill n01833805 hummingbird n01843065 jacamar n01843383 toucan n01847000 drake n01855032 red-breasted merganser, Mergus serrator n01855672 goose n01860187 black swan, Cygnus atratus n01871265 tusker n01872401 echidna, spiny anteater, anteater n01873310 platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus n01877812 wallaby, brush kangaroo n01882714 koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus n01883070 wombat n01910747 jellyfish n01914609 sea anemone, anemone n01917289 brain coral n01924916 flatworm, platyhelminth n01930112 nematode, nematode worm, roundworm n01943899 conch n01944390 snail n01945685 slug n01950731 sea slug, nudibranch n01955084 chiton, coat-of-mail shell, sea cradle, polyplacophore n01968897 chambered nautilus, pearly nautilus, nautilus n01978287 Dungeness crab, Cancer magister n01978455 rock crab, Cancer irroratus n01980166 fiddler crab n01981276 king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica n01983481 American lobster, Northern lobster, Maine lobster, Homarus americanus n01984695 spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish n01985128 crayfish, crawfish, crawdad, crawdaddy n01986214 hermit crab n01990800 isopod n02002556 white stork, Ciconia ciconia n02002724 black stork, Ciconia nigra n02006656 spoonbill n02007558 flamingo n02009229 little blue heron, Egretta caerulea n02009912 American egret, great white heron, Egretta albus n02011460 bittern n02012849 crane n02013706 limpkin, Aramus pictus n02017213 European gallinule, Porphyrio porphyrio n02018207 American coot, marsh hen, mud hen, water hen, Fulica americana n02018795 bustard n02025239 ruddy turnstone, Arenaria interpres n02027492 red-backed sandpiper, dunlin, Erolia alpina n02028035 redshank, Tringa totanus n02033041 dowitcher n02037110 oystercatcher, oyster catcher n02051845 pelican n02056570 king penguin, Aptenodytes patagonica n02058221 albatross, mollymawk n02066245 grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus n02071294 killer whale, killer, orca, grampus, sea wolf, Orcinus orca n02074367 dugong, Dugong dugon n02077923 sea lion n02085620 Chihuahua n02085782 Japanese spaniel n02085936 Maltese dog, Maltese terrier, Maltese n02086079 Pekinese, Pekingese, Peke n02086240 Shih-Tzu n02086646 Blenheim spaniel n02086910 papillon n02087046 toy terrier n02087394 Rhodesian ridgeback n02088094 Afghan hound, Afghan n02088238 basset, basset hound n02088364 beagle n02088466 bloodhound, sleuthhound n02088632 bluetick n02089078 black-and-tan coonhound n02089867 Walker hound, Walker foxhound n02089973 English foxhound n02090379 redbone n02090622 borzoi, Russian wolfhound n02090721 Irish wolfhound n02091032 Italian greyhound n02091134 whippet n02091244 Ibizan hound, Ibizan Podenco n02091467 Norwegian elkhound, elkhound n02091635 otterhound, otter hound n02091831 Saluki, gazelle hound n02092002 Scottish deerhound, deerhound n02092339 Weimaraner n02093256 Staffordshire bullterrier, Staffordshire bull terrier n02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier n02093647 Bedlington terrier n02093754 Border terrier n02093859 Kerry blue terrier n02093991 Irish terrier n02094114 Norfolk terrier n02094258 Norwich terrier n02094433 Yorkshire terrier n02095314 wire-haired fox terrier n02095570 Lakeland terrier n02095889 Sealyham terrier, Sealyham n02096051 Airedale, Airedale terrier n02096177 cairn, cairn terrier n02096294 Australian terrier n02096437 Dandie Dinmont, Dandie Dinmont terrier n02096585 Boston bull, Boston terrier n02097047 miniature schnauzer n02097130 giant schnauzer n02097209 standard schnauzer n02097298 Scotch terrier, Scottish terrier, Scottie n02097474 Tibetan terrier, chrysanthemum dog n02097658 silky terrier, Sydney silky n02098105 soft-coated wheaten terrier n02098286 West Highland white terrier n02098413 Lhasa, Lhasa apso n02099267 flat-coated retriever n02099429 curly-coated retriever n02099601 golden retriever n02099712 Labrador retriever n02099849 Chesapeake Bay retriever n02100236 German short-haired pointer n02100583 vizsla, Hungarian pointer n02100735 English setter n02100877 Irish setter, red setter n02101006 Gordon setter n02101388 Brittany spaniel n02101556 clumber, clumber spaniel n02102040 English springer, English springer spaniel n02102177 Welsh springer spaniel n02102318 cocker spaniel, English cocker spaniel, cocker n02102480 Sussex spaniel n02102973 Irish water spaniel n02104029 kuvasz n02104365 schipperke n02105056 groenendael n02105162 malinois n02105251 briard n02105412 kelpie n02105505 komondor n02105641 Old English sheepdog, bobtail n02105855 Shetland sheepdog, Shetland sheep dog, Shetland n02106030 collie n02106166 Border collie n02106382 Bouvier des Flandres, Bouviers des Flandres n02106550 Rottweiler n02106662 German shepherd, German shepherd dog, German police dog, alsatian n02107142 Doberman, Doberman pinscher n02107312 miniature pinscher n02107574 Greater Swiss Mountain dog n02107683 Bernese mountain dog n02107908 Appenzeller n02108000 EntleBucher n02108089 boxer n02108422 bull mastiff n02108551 Tibetan mastiff n02108915 French bulldog n02109047 Great Dane n02109525 Saint Bernard, St Bernard n02109961 Eskimo dog, husky n02110063 malamute, malemute, Alaskan malamute n02110185 Siberian husky n02110341 dalmatian, coach dog, carriage dog n02110627 affenpinscher, monkey pinscher, monkey dog n02110806 basenji n02110958 pug, pug-dog n02111129 Leonberg n02111277 Newfoundland, Newfoundland dog n02111500 Great Pyrenees n02111889 Samoyed, Samoyede n02112018 Pomeranian n02112137 chow, chow chow n02112350 keeshond n02112706 Brabancon griffon n02113023 Pembroke, Pembroke Welsh corgi n02113186 Cardigan, Cardigan Welsh corgi n02113624 toy poodle n02113712 miniature poodle n02113799 standard poodle n02113978 Mexican hairless n02114367 timber wolf, grey wolf, gray wolf, Canis lupus n02114548 white wolf, Arctic wolf, Canis lupus tundrarum n02114712 red wolf, maned wolf, Canis rufus, Canis niger n02114855 coyote, prairie wolf, brush wolf, Canis latrans n02115641 dingo, warrigal, warragal, Canis dingo n02115913 dhole, Cuon alpinus n02116738 African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus n02117135 hyena, hyaena n02119022 red fox, Vulpes vulpes n02119789 kit fox, Vulpes macrotis n02120079 Arctic fox, white fox, Alopex lagopus n02120505 grey fox, gray fox, Urocyon cinereoargenteus n02123045 tabby, tabby cat n02123159 tiger cat n02123394 Persian cat n02123597 Siamese cat, Siamese n02124075 Egyptian cat n02125311 cougar, puma, catamount, mountain lion, painter, panther, Felis concolor n02127052 lynx, catamount n02128385 leopard, Panthera pardus n02128757 snow leopard, ounce, Panthera uncia n02128925 jaguar, panther, Panthera onca, Felis onca n02129165 lion, king of beasts, Panthera leo n02129604 tiger, Panthera tigris n02130308 cheetah, chetah, Acinonyx jubatus n02132136 brown bear, bruin, Ursus arctos n02133161 American black bear, black bear, Ursus americanus, Euarctos americanus n02134084 ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus n02134418 sloth bear, Melursus ursinus, Ursus ursinus n02137549 mongoose n02138441 meerkat, mierkat n02165105 tiger beetle n02165456 ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle n02167151 ground beetle, carabid beetle n02168699 long-horned beetle, longicorn, longicorn beetle n02169497 leaf beetle, chrysomelid n02172182 dung beetle n02174001 rhinoceros beetle n02177972 weevil n02190166 fly n02206856 bee n02219486 ant, emmet, pismire n02226429 grasshopper, hopper n02229544 cricket n02231487 walking stick, walkingstick, stick insect n02233338 cockroach, roach n02236044 mantis, mantid n02256656 cicada, cicala n02259212 leafhopper n02264363 lacewing, lacewing fly n02268443 dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk n02268853 damselfly n02276258 admiral n02277742 ringlet, ringlet butterfly n02279972 monarch, monarch butterfly, milkweed butterfly, Danaus plexippus n02280649 cabbage butterfly n02281406 sulphur butterfly, sulfur butterfly n02281787 lycaenid, lycaenid butterfly n02317335 starfish, sea star n02319095 sea urchin n02321529 sea cucumber, holothurian n02325366 wood rabbit, cottontail, cottontail rabbit n02326432 hare n02328150 Angora, Angora rabbit n02342885 hamster n02346627 porcupine, hedgehog n02356798 fox squirrel, eastern fox squirrel, Sciurus niger n02361337 marmot n02363005 beaver n02364673 guinea pig, Cavia cobaya n02389026 sorrel n02391049 zebra n02395406 hog, pig, grunter, squealer, Sus scrofa n02396427 wild boar, boar, Sus scrofa n02397096 warthog n02398521 hippopotamus, hippo, river horse, Hippopotamus amphibius n02403003 ox n02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis n02410509 bison n02412080 ram, tup n02415577 bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis n02417914 ibex, Capra ibex n02422106 hartebeest n02422699 impala, Aepyceros melampus n02423022 gazelle n02437312 Arabian camel, dromedary, Camelus dromedarius n02437616 llama n02441942 weasel n02442845 mink n02443114 polecat, fitch, foulmart, foumart, Mustela putorius n02443484 black-footed ferret, ferret, Mustela nigripes n02444819 otter n02445715 skunk, polecat, wood pussy n02447366 badger n02454379 armadillo n02457408 three-toed sloth, ai, Bradypus tridactylus n02480495 orangutan, orang, orangutang, Pongo pygmaeus n02480855 gorilla, Gorilla gorilla n02481823 chimpanzee, chimp, Pan troglodytes n02483362 gibbon, Hylobates lar n02483708 siamang, Hylobates syndactylus, Symphalangus syndactylus n02484975 guenon, guenon monkey n02486261 patas, hussar monkey, Erythrocebus patas n02486410 baboon n02487347 macaque n02488291 langur n02488702 colobus, colobus monkey n02489166 proboscis monkey, Nasalis larvatus n02490219 marmoset n02492035 capuchin, ringtail, Cebus capucinus n02492660 howler monkey, howler n02493509 titi, titi monkey n02493793 spider monkey, Ateles geoffroyi n02494079 squirrel monkey, Saimiri sciureus n02497673 Madagascar cat, ring-tailed lemur, Lemur catta n02500267 indri, indris, Indri indri, Indri brevicaudatus n02504013 Indian elephant, Elephas maximus n02504458 African elephant, Loxodonta africana n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca n02514041 barracouta, snoek n02526121 eel n02536864 coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch n02606052 rock beauty, Holocanthus tricolor n02607072 anemone fish n02640242 sturgeon n02641379 gar, garfish, garpike, billfish, Lepisosteus osseus n02643566 lionfish n02655020 puffer, pufferfish, blowfish, globefish n02666196 abacus n02667093 abaya n02669723 academic gown, academic robe, judge's robe n02672831 accordion, piano accordion, squeeze box n02676566 acoustic guitar n02687172 aircraft carrier, carrier, flattop, attack aircraft carrier n02690373 airliner n02692877 airship, dirigible n02699494 altar n02701002 ambulance n02704792 amphibian, amphibious vehicle n02708093 analog clock n02727426 apiary, bee house n02730930 apron n02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin n02749479 assault rifle, assault gun n02769748 backpack, back pack, knapsack, packsack, rucksack, haversack n02776631 bakery, bakeshop, bakehouse n02777292 balance beam, beam n02782093 balloon n02783161 ballpoint, ballpoint pen, ballpen, Biro n02786058 Band Aid n02787622 banjo n02788148 bannister, banister, balustrade, balusters, handrail n02790996 barbell n02791124 barber chair n02791270 barbershop n02793495 barn n02794156 barometer n02795169 barrel, cask n02797295 barrow, garden cart, lawn cart, wheelbarrow n02799071 baseball n02802426 basketball n02804414 bassinet n02804610 bassoon n02807133 bathing cap, swimming cap n02808304 bath towel n02808440 bathtub, bathing tub, bath, tub n02814533 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon n02814860 beacon, lighthouse, beacon light, pharos n02815834 beaker n02817516 bearskin, busby, shako n02823428 beer bottle n02823750 beer glass n02825657 bell cote, bell cot n02834397 bib n02835271 bicycle-built-for-two, tandem bicycle, tandem n02837789 bikini, two-piece n02840245 binder, ring-binder n02841315 binoculars, field glasses, opera glasses n02843684 birdhouse n02859443 boathouse n02860847 bobsled, bobsleigh, bob n02865351 bolo tie, bolo, bola tie, bola n02869837 bonnet, poke bonnet n02870880 bookcase n02871525 bookshop, bookstore, bookstall n02877765 bottlecap n02879718 bow n02883205 bow tie, bow-tie, bowtie n02892201 brass, memorial tablet, plaque n02892767 brassiere, bra, bandeau n02894605 breakwater, groin, groyne, mole, bulwark, seawall, jetty n02895154 breastplate, aegis, egis n02906734 broom n02909870 bucket, pail n02910353 buckle n02916936 bulletproof vest n02917067 bullet train, bullet n02927161 butcher shop, meat market n02930766 cab, hack, taxi, taxicab n02939185 caldron, cauldron n02948072 candle, taper, wax light n02950826 cannon n02951358 canoe n02951585 can opener, tin opener n02963159 cardigan n02965783 car mirror n02966193 carousel, carrousel, merry-go-round, roundabout, whirligig n02966687 carpenter's kit, tool kit n02971356 carton n02974003 car wheel n02977058 cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM n02978881 cassette n02979186 cassette player n02980441 castle n02981792 catamaran n02988304 CD player n02992211 cello, violoncello n02992529 cellular telephone, cellular phone, cellphone, cell, mobile phone n02999410 chain n03000134 chainlink fence n03000247 chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour n03000684 chain saw, chainsaw n03014705 chest n03016953 chiffonier, commode n03017168 chime, bell, gong n03018349 china cabinet, china closet n03026506 Christmas stocking n03028079 church, church building n03032252 cinema, movie theater, movie theatre, movie house, picture palace n03041632 cleaver, meat cleaver, chopper n03042490 cliff dwelling n03045698 cloak n03047690 clog, geta, patten, sabot n03062245 cocktail shaker n03063599 coffee mug n03063689 coffeepot n03065424 coil, spiral, volute, whorl, helix n03075370 combination lock n03085013 computer keyboard, keypad n03089624 confectionery, confectionary, candy store n03095699 container ship, containership, container vessel n03100240 convertible n03109150 corkscrew, bottle screw n03110669 cornet, horn, trumpet, trump n03124043 cowboy boot n03124170 cowboy hat, ten-gallon hat n03125729 cradle n03126707 crane n03127747 crash helmet n03127925 crate n03131574 crib, cot n03133878 Crock Pot n03134739 croquet ball n03141823 crutch n03146219 cuirass n03160309 dam, dike, dyke n03179701 desk n03180011 desktop computer n03187595 dial telephone, dial phone n03188531 diaper, nappy, napkin n03196217 digital clock n03197337 digital watch n03201208 dining table, board n03207743 dishrag, dishcloth n03207941 dishwasher, dish washer, dishwashing machine n03208938 disk brake, disc brake n03216828 dock, dockage, docking facility n03218198 dogsled, dog sled, dog sleigh n03220513 dome n03223299 doormat, welcome mat n03240683 drilling platform, offshore rig n03249569 drum, membranophone, tympan n03250847 drumstick n03255030 dumbbell n03259280 Dutch oven n03271574 electric fan, blower n03272010 electric guitar n03272562 electric locomotive n03290653 entertainment center n03291819 envelope n03297495 espresso maker n03314780 face powder n03325584 feather boa, boa n03337140 file, file cabinet, filing cabinet n03344393 fireboat n03345487 fire engine, fire truck n03347037 fire screen, fireguard n03355925 flagpole, flagstaff n03372029 flute, transverse flute n03376595 folding chair n03379051 football helmet n03384352 forklift n03388043 fountain n03388183 fountain pen n03388549 four-poster n03393912 freight car n03394916 French horn, horn n03400231 frying pan, frypan, skillet n03404251 fur coat n03417042 garbage truck, dustcart n03424325 gasmask, respirator, gas helmet n03425413 gas pump, gasoline pump, petrol pump, island dispenser n03443371 goblet n03444034 go-kart n03445777 golf ball n03445924 golfcart, golf cart n03447447 gondola n03447721 gong, tam-tam n03450230 gown n03452741 grand piano, grand n03457902 greenhouse, nursery, glasshouse n03459775 grille, radiator grille n03461385 grocery store, grocery, food market, market n03467068 guillotine n03476684 hair slide n03476991 hair spray n03478589 half track n03481172 hammer n03482405 hamper n03483316 hand blower, blow dryer, blow drier, hair dryer, hair drier n03485407 hand-held computer, hand-held microcomputer n03485794 handkerchief, hankie, hanky, hankey n03492542 hard disc, hard disk, fixed disk n03494278 harmonica, mouth organ, harp, mouth harp n03495258 harp n03496892 harvester, reaper n03498962 hatchet n03527444 holster n03529860 home theater, home theatre n03530642 honeycomb n03532672 hook, claw n03534580 hoopskirt, crinoline n03535780 horizontal bar, high bar n03538406 horse cart, horse-cart n03544143 hourglass n03584254 iPod n03584829 iron, smoothing iron n03590841 jack-o'-lantern n03594734 jean, blue jean, denim n03594945 jeep, landrover n03595614 jersey, T-shirt, tee shirt n03598930 jigsaw puzzle n03599486 jinrikisha, ricksha, rickshaw n03602883 joystick n03617480 kimono n03623198 knee pad n03627232 knot n03630383 lab coat, laboratory coat n03633091 ladle n03637318 lampshade, lamp shade n03642806 laptop, laptop computer n03649909 lawn mower, mower n03657121 lens cap, lens cover n03658185 letter opener, paper knife, paperknife n03661043 library n03662601 lifeboat n03666591 lighter, light, igniter, ignitor n03670208 limousine, limo n03673027 liner, ocean liner n03676483 lipstick, lip rouge n03680355 Loafer n03690938 lotion n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system n03692522 loupe, jeweler's loupe n03697007 lumbermill, sawmill n03706229 magnetic compass n03709823 mailbag, postbag n03710193 mailbox, letter box n03710637 maillot n03710721 maillot, tank suit n03717622 manhole cover n03720891 maraca n03721384 marimba, xylophone n03724870 mask n03729826 matchstick n03733131 maypole n03733281 maze, labyrinth n03733805 measuring cup n03742115 medicine chest, medicine cabinet n03743016 megalith, megalithic structure n03759954 microphone, mike n03761084 microwave, microwave oven n03763968 military uniform n03764736 milk can n03769881 minibus n03770439 miniskirt, mini n03770679 minivan n03773504 missile n03775071 mitten n03775546 mixing bowl n03776460 mobile home, manufactured home n03777568 Model T n03777754 modem n03781244 monastery n03782006 monitor n03785016 moped n03786901 mortar n03787032 mortarboard n03788195 mosque n03788365 mosquito net n03791053 motor scooter, scooter n03792782 mountain bike, all-terrain bike, off-roader n03792972 mountain tent n03793489 mouse, computer mouse n03794056 mousetrap n03796401 moving van n03803284 muzzle n03804744 nail n03814639 neck brace n03814906 necklace n03825788 nipple n03832673 notebook, notebook computer n03837869 obelisk n03838899 oboe, hautboy, hautbois n03840681 ocarina, sweet potato n03841143 odometer, hodometer, mileometer, milometer n03843555 oil filter n03854065 organ, pipe organ n03857828 oscilloscope, scope, cathode-ray oscilloscope, CRO n03866082 overskirt n03868242 oxcart n03868863 oxygen mask n03871628 packet n03873416 paddle, boat paddle n03874293 paddlewheel, paddle wheel n03874599 padlock n03876231 paintbrush n03877472 pajama, pyjama, pj's, jammies n03877845 palace n03884397 panpipe, pandean pipe, syrinx n03887697 paper towel n03888257 parachute, chute n03888605 parallel bars, bars n03891251 park bench n03891332 parking meter n03895866 passenger car, coach, carriage n03899768 patio, terrace n03902125 pay-phone, pay-station n03903868 pedestal, plinth, footstall n03908618 pencil box, pencil case n03908714 pencil sharpener n03916031 perfume, essence n03920288 Petri dish n03924679 photocopier n03929660 pick, plectrum, plectron n03929855 pickelhaube n03930313 picket fence, paling n03930630 pickup, pickup truck n03933933 pier n03935335 piggy bank, penny bank n03937543 pill bottle n03938244 pillow n03942813 ping-pong ball n03944341 pinwheel n03947888 pirate, pirate ship n03950228 pitcher, ewer n03954731 plane, carpenter's plane, woodworking plane n03956157 planetarium n03958227 plastic bag n03961711 plate rack n03967562 plow, plough n03970156 plunger, plumber's helper n03976467 Polaroid camera, Polaroid Land camera n03976657 pole n03977966 police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria n03980874 poncho n03982430 pool table, billiard table, snooker table n03983396 pop bottle, soda bottle n03991062 pot, flowerpot n03992509 potter's wheel n03995372 power drill n03998194 prayer rug, prayer mat n04004767 printer n04005630 prison, prison house n04008634 projectile, missile n04009552 projector n04019541 puck, hockey puck n04023962 punching bag, punch bag, punching ball, punchball n04026417 purse n04033901 quill, quill pen n04033995 quilt, comforter, comfort, puff n04037443 racer, race car, racing car n04039381 racket, racquet n04040759 radiator n04041544 radio, wireless n04044716 radio telescope, radio reflector n04049303 rain barrel n04065272 recreational vehicle, RV, R.V. n04067472 reel n04069434 reflex camera n04070727 refrigerator, icebox n04074963 remote control, remote n04081281 restaurant, eating house, eating place, eatery n04086273 revolver, six-gun, six-shooter n04090263 rifle n04099969 rocking chair, rocker n04111531 rotisserie n04116512 rubber eraser, rubber, pencil eraser n04118538 rugby ball n04118776 rule, ruler n04120489 running shoe n04125021 safe n04127249 safety pin n04131690 saltshaker, salt shaker n04133789 sandal n04136333 sarong n04141076 sax, saxophone n04141327 scabbard n04141975 scale, weighing machine n04146614 school bus n04147183 schooner n04149813 scoreboard n04152593 screen, CRT screen n04153751 screw n04154565 screwdriver n04162706 seat belt, seatbelt n04179913 sewing machine n04192698 shield, buckler n04200800 shoe shop, shoe-shop, shoe store n04201297 shoji n04204238 shopping basket n04204347 shopping cart n04208210 shovel n04209133 shower cap n04209239 shower curtain n04228054 ski n04229816 ski mask n04235860 sleeping bag n04238763 slide rule, slipstick n04239074 sliding door n04243546 slot, one-armed bandit n04251144 snorkel n04252077 snowmobile n04252225 snowplow, snowplough n04254120 soap dispenser n04254680 soccer ball n04254777 sock n04258138 solar dish, solar collector, solar furnace n04259630 sombrero n04263257 soup bowl n04264628 space bar n04265275 space heater n04266014 space shuttle n04270147 spatula n04273569 speedboat n04275548 spider web, spider's web n04277352 spindle n04285008 sports car, sport car n04286575 spotlight, spot n04296562 stage n04310018 steam locomotive n04311004 steel arch bridge n04311174 steel drum n04317175 stethoscope n04325704 stole n04326547 stone wall n04328186 stopwatch, stop watch n04330267 stove n04332243 strainer n04335435 streetcar, tram, tramcar, trolley, trolley car n04336792 stretcher n04344873 studio couch, day bed n04346328 stupa, tope n04347754 submarine, pigboat, sub, U-boat n04350905 suit, suit of clothes n04355338 sundial n04355933 sunglass n04356056 sunglasses, dark glasses, shades n04357314 sunscreen, sunblock, sun blocker n04366367 suspension bridge n04367480 swab, swob, mop n04370456 sweatshirt n04371430 swimming trunks, bathing trunks n04371774 swing n04372370 switch, electric switch, electrical switch n04376876 syringe n04380533 table lamp n04389033 tank, army tank, armored combat vehicle, armoured combat vehicle n04392985 tape player n04398044 teapot n04399382 teddy, teddy bear n04404412 television, television system n04409515 tennis ball n04417672 thatch, thatched roof n04418357 theater curtain, theatre curtain n04423845 thimble n04428191 thresher, thrasher, threshing machine n04429376 throne n04435653 tile roof n04442312 toaster n04443257 tobacco shop, tobacconist shop, tobacconist n04447861 toilet seat n04456115 torch n04458633 totem pole n04461696 tow truck, tow car, wrecker n04462240 toyshop n04465501 tractor n04467665 trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi n04476259 tray n04479046 trench coat n04482393 tricycle, trike, velocipede n04483307 trimaran n04485082 tripod n04486054 triumphal arch n04487081 trolleybus, trolley coach, trackless trolley n04487394 trombone n04493381 tub, vat n04501370 turnstile n04505470 typewriter keyboard n04507155 umbrella n04509417 unicycle, monocycle n04515003 upright, upright piano n04517823 vacuum, vacuum cleaner n04522168 vase n04523525 vault n04525038 velvet n04525305 vending machine n04532106 vestment n04532670 viaduct n04536866 violin, fiddle n04540053 volleyball n04542943 waffle iron n04548280 wall clock n04548362 wallet, billfold, notecase, pocketbook n04550184 wardrobe, closet, press n04552348 warplane, military plane n04553703 washbasin, handbasin, washbowl, lavabo, wash-hand basin n04554684 washer, automatic washer, washing machine n04557648 water bottle n04560804 water jug n04562935 water tower n04579145 whiskey jug n04579432 whistle n04584207 wig n04589890 window screen n04590129 window shade n04591157 Windsor tie n04591713 wine bottle n04592741 wing n04596742 wok n04597913 wooden spoon n04599235 wool, woolen, woollen n04604644 worm fence, snake fence, snake-rail fence, Virginia fence n04606251 wreck n04612504 yawl n04613696 yurt n06359193 web site, website, internet site, site n06596364 comic book n06785654 crossword puzzle, crossword n06794110 street sign n06874185 traffic light, traffic signal, stoplight n07248320 book jacket, dust cover, dust jacket, dust wrapper n07565083 menu n07579787 plate n07583066 guacamole n07584110 consomme n07590611 hot pot, hotpot n07613480 trifle n07614500 ice cream, icecream n07615774 ice lolly, lolly, lollipop, popsicle n07684084 French loaf n07693725 bagel, beigel n07695742 pretzel n07697313 cheeseburger n07697537 hotdog, hot dog, red hot n07711569 mashed potato n07714571 head cabbage n07714990 broccoli n07715103 cauliflower n07716358 zucchini, courgette n07716906 spaghetti squash n07717410 acorn squash n07717556 butternut squash n07718472 cucumber, cuke n07718747 artichoke, globe artichoke n07720875 bell pepper n07730033 cardoon n07734744 mushroom n07742313 Granny Smith n07745940 strawberry n07747607 orange n07749582 lemon n07753113 fig n07753275 pineapple, ananas n07753592 banana n07754684 jackfruit, jak, jack n07760859 custard apple n07768694 pomegranate n07802026 hay n07831146 carbonara n07836838 chocolate sauce, chocolate syrup n07860988 dough n07871810 meat loaf, meatloaf n07873807 pizza, pizza pie n07875152 potpie n07880968 burrito n07892512 red wine n07920052 espresso n07930864 cup n07932039 eggnog n09193705 alp n09229709 bubble n09246464 cliff, drop, drop-off n09256479 coral reef n09288635 geyser n09332890 lakeside, lakeshore n09399592 promontory, headland, head, foreland n09421951 sandbar, sand bar n09428293 seashore, coast, seacoast, sea-coast n09468604 valley, vale n09472597 volcano n09835506 ballplayer, baseball player n10148035 groom, bridegroom n10565667 scuba diver n11879895 rapeseed n11939491 daisy n12057211 yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum n12144580 corn n12267677 acorn n12620546 hip, rose hip, rosehip n12768682 buckeye, horse chestnut, conker n12985857 coral fungus n12998815 agaric n13037406 gyromitra n13040303 stinkhorn, carrion fungus n13044778 earthstar n13052670 hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa n13054560 bolete n13133613 ear, spike, capitulum n15075141 toilet tissue, toilet paper, bathroom tissue ================================================ FILE: li_attack.py ================================================ ## li_attack.py -- attack a network optimizing for l_infinity distance ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import sys import tensorflow as tf import numpy as np DECREASE_FACTOR = 0.9 # 01, rate at which we increase constant, smaller better class CarliniLi: def __init__(self, sess, model, targeted = TARGETED, learning_rate = LEARNING_RATE, max_iterations = MAX_ITERATIONS, abort_early = ABORT_EARLY, initial_const = INITIAL_CONST, largest_const = LARGEST_CONST, reduce_const = REDUCE_CONST, decrease_factor = DECREASE_FACTOR, const_factor = CONST_FACTOR): """ The L_infinity optimized attack. Returns adversarial examples for the supplied model. targeted: True if we should perform a targetted attack, False otherwise. learning_rate: The learning rate for the attack algorithm. Smaller values produce better results but are slower to converge. max_iterations: The maximum number of iterations. Larger values are more accurate; setting too small will require a large learning rate and will produce poor results. abort_early: If true, allows early aborts if gradient descent gets stuck. initial_const: The initial tradeoff-constant to use to tune the relative importance of distance and confidence. Should be set to a very small value (but positive). largest_const: The largest constant to use until we report failure. Should be set to a very large value. reduce_const: If true, after each successful attack, make const smaller. decrease_factor: Rate at which we should decrease tau, less than one. Larger produces better quality results. const_factor: The rate at which we should increase the constant, when the previous constant failed. Should be greater than one, smaller is better. """ self.model = model self.sess = sess self.TARGETED = targeted self.LEARNING_RATE = learning_rate self.MAX_ITERATIONS = max_iterations self.ABORT_EARLY = abort_early self.INITIAL_CONST = initial_const self.LARGEST_CONST = largest_const self.DECREASE_FACTOR = decrease_factor self.REDUCE_CONST = reduce_const self.const_factor = const_factor self.grad = self.gradient_descent(sess, model) def gradient_descent(self, sess, model): def compare(x,y): if self.TARGETED: return x == y else: return x != y shape = (1,model.image_size,model.image_size,model.num_channels) # the variable to optimize over modifier = tf.Variable(np.zeros(shape,dtype=np.float32)) tau = tf.placeholder(tf.float32, []) simg = tf.placeholder(tf.float32, shape) timg = tf.placeholder(tf.float32, shape) tlab = tf.placeholder(tf.float32, (1,model.num_labels)) const = tf.placeholder(tf.float32, []) newimg = (tf.tanh(modifier + simg)/2) output = model.predict(newimg) orig_output = model.predict(tf.tanh(timg)/2) real = tf.reduce_sum((tlab)*output) other = tf.reduce_max((1-tlab)*output - (tlab*10000)) if self.TARGETED: # if targetted, optimize for making the other class most likely loss1 = tf.maximum(0.0,other-real) else: # if untargeted, optimize for making this class least likely. loss1 = tf.maximum(0.0,real-other) # sum up the losses loss2 = tf.reduce_sum(tf.maximum(0.0,tf.abs(newimg-tf.tanh(timg)/2)-tau)) loss = const*loss1+loss2 # setup the adam optimizer and keep track of variables we're creating start_vars = set(x.name for x in tf.global_variables()) optimizer = tf.train.AdamOptimizer(self.LEARNING_RATE) train = optimizer.minimize(loss, var_list=[modifier]) end_vars = tf.global_variables() new_vars = [x for x in end_vars if x.name not in start_vars] init = tf.variables_initializer(var_list=[modifier]+new_vars) def doit(oimgs, labs, starts, tt, CONST): # convert to tanh-space imgs = np.arctanh(np.array(oimgs)*1.999999) starts = np.arctanh(np.array(starts)*1.999999) # initialize the variables sess.run(init) while CONST < self.LARGEST_CONST: # try solving for each value of the constant print('try const', CONST) for step in range(self.MAX_ITERATIONS): feed_dict={timg: imgs, tlab:labs, tau: tt, simg: starts, const: CONST} if step%(self.MAX_ITERATIONS//10) == 0: print(step,sess.run((loss,loss1,loss2),feed_dict=feed_dict)) # perform the update step _, works = sess.run([train, loss], feed_dict=feed_dict) # it worked if works < .0001*CONST and (self.ABORT_EARLY or step == CONST-1): get = sess.run(output, feed_dict=feed_dict) works = compare(np.argmax(get), np.argmax(labs)) if works: scores, origscores, nimg = sess.run((output,orig_output,newimg),feed_dict=feed_dict) l2s=np.square(nimg-np.tanh(imgs)/2).sum(axis=(1,2,3)) return scores, origscores, nimg, CONST # we didn't succeed, increase constant and try again CONST *= self.const_factor return doit def attack(self, imgs, targets): """ Perform the L_0 attack on the given images for the given targets. If self.targeted is true, then the targets represents the target labels. If self.targeted is false, then targets are the original class labels. """ r = [] for img,target in zip(imgs, targets): r.extend(self.attack_single(img, target)) return np.array(r) def attack_single(self, img, target): """ Run the attack on a single image and label """ # the previous image prev = np.copy(img).reshape((1,self.model.image_size,self.model.image_size,self.model.num_channels)) tau = 1.0 const = self.INITIAL_CONST while tau > 1./256: # try to solve given this tau value res = self.grad([np.copy(img)], [target], np.copy(prev), tau, const) if res == None: # the attack failed, we return this as our final answer return prev scores, origscores, nimg, const = res if self.REDUCE_CONST: const /= 2 # the attack succeeded, reduce tau and try again actualtau = np.max(np.abs(nimg-img)) if actualtau < tau: tau = actualtau print("Tau",tau) prev = nimg tau *= self.DECREASE_FACTOR return prev ================================================ FILE: mnist_blackbox.py ================================================ ## Copyright (C) IBM Corp, 2017-2018 from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals import os import time import numpy as np from six.moves import xrange import keras from keras import backend from keras.utils.np_utils import to_categorical from keras.models import Sequential from keras.layers import Dense, Flatten, Activation, Dropout import tensorflow as tf from tensorflow.python.platform import app from tensorflow.python.platform import flags from cleverhans.utils_keras import cnn_model from cleverhans.utils_mnist import data_mnist from cleverhans.utils_tf import model_train, model_eval, batch_eval, tf_model_load from cleverhans.attacks import CarliniWagnerL2 from cleverhans.attacks import FastGradientMethod from cleverhans.attacks_tf import jacobian_graph, jacobian_augmentation from cleverhans.utils_keras import KerasModelWrapper from setup_mnist import MNISTModel FLAGS = flags.FLAGS def setup_tutorial(): """ Helper function to check correct configuration of tf and keras for tutorial :return: True if setup checks completed """ # Set TF random seed to improve reproducibility tf.set_random_seed(1234) if not hasattr(backend, "tf"): raise RuntimeError("This tutorial requires keras to be configured" " to use the TensorFlow backend.") # Image dimensions ordering should follow the Theano convention if keras.backend.image_dim_ordering() != 'tf': keras.backend.set_image_dim_ordering('tf') print("INFO: '~/.keras/keras.json' sets 'image_dim_ordering' " "to 'th', temporarily setting to 'tf'") return True def prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate): """ Define and train a model that simulates the "remote" black-box oracle described in the original paper. :param sess: the TF session :param x: the input placeholder for MNIST :param y: the ouput placeholder for MNIST :param X_train: the training data for the oracle :param Y_train: the training labels for the oracle :param X_test: the testing data for the oracle :param Y_test: the testing labels for the oracle :param nb_epochs: number of epochs to train model :param batch_size: size of training batches :param learning_rate: learning rate for training :return: """ # Define TF model graph (for the black-box model) model = MNISTModel(use_log = True).model predictions = model(x) print("Defined TensorFlow model graph.") # Train an MNIST model if FLAGS.load_pretrain: tf_model_load(sess) else: train_params = { 'nb_epochs': nb_epochs, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, predictions, X_train, Y_train, verbose=False, save=True, args=train_params) # Print out the accuracy on legitimate data eval_params = {'batch_size': batch_size} accuracy = model_eval(sess, x, y, predictions, X_test, Y_test, args=eval_params) print('Test accuracy of black-box on legitimate test ' 'examples: ' + str(accuracy)) return model, predictions, accuracy def substitute_model(img_rows=28, img_cols=28, nb_classes=10): """ Defines the model architecture to be used by the substitute :param img_rows: number of rows in input :param img_cols: number of columns in input :param nb_classes: number of classes in output :return: keras model """ model = Sequential() # Find out the input shape ordering if keras.backend.image_dim_ordering() == 'th': input_shape = (1, img_rows, img_cols) else: input_shape = (img_rows, img_cols, 1) # Define a fully connected model (it's different than the black-box) layers = [Flatten(input_shape=input_shape), Dense(200), Activation('relu'), Dropout(0.5), Dense(200), Activation('relu'), Dropout(0.5), Dense(nb_classes), Activation('softmax')] for layer in layers: model.add(layer) return model def train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda): """ This function creates the substitute by alternatively augmenting the training data and training the substitute. :param sess: TF session :param x: input TF placeholder :param y: output TF placeholder :param bbox_preds: output of black-box model predictions :param X_sub: initial substitute training data :param Y_sub: initial substitute training labels :param nb_classes: number of output classes :param nb_epochs_s: number of epochs to train substitute model :param batch_size: size of training batches :param learning_rate: learning rate for training :param data_aug: number of times substitute training data is augmented :param lmbda: lambda from arxiv.org/abs/1602.02697 :return: """ # Define TF model graph (for the black-box model) # model_sub = substitute_model() model_sub = MNISTModel(use_log = True).model preds_sub = model_sub(x) print("Defined TensorFlow model graph for the substitute.") # Define the Jacobian symbolically using TensorFlow grads = jacobian_graph(preds_sub, x, nb_classes) # Train the substitute and augment dataset alternatively for rho in xrange(data_aug): print("Substitute training epoch #" + str(rho)) train_params = { 'nb_epochs': nb_epochs_s, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, preds_sub, X_sub, to_categorical(Y_sub), init_all=False, verbose=False, args=train_params) # If we are not at last substitute training iteration, augment dataset if rho < data_aug - 1: if FLAGS.cached_aug: augs = np.load('sub_saved/mnist-aug-{}.npz'.format(rho)) X_sub = augs['X_sub'] Y_sub = augs['Y_sub'] else: print("Augmenting substitute training data.") # Perform the Jacobian augmentation X_sub = jacobian_augmentation(sess, x, X_sub, Y_sub, grads, lmbda) print("Labeling substitute training data.") # Label the newly generated synthetic points using the black-box Y_sub = np.hstack([Y_sub, Y_sub]) X_sub_prev = X_sub[int(len(X_sub)/2):] eval_params = {'batch_size': batch_size} bbox_val = batch_eval(sess, [x], [bbox_preds], [X_sub_prev], args=eval_params)[0] # Note here that we take the argmax because the adversary # only has access to the label (not the probabilities) output # by the black-box model Y_sub[int(len(X_sub)/2):] = np.argmax(bbox_val, axis=1) # cache the augmentation if not FLAGS.cached_aug: np.savez('sub_saved/mnist-aug-{}.npz'.format(rho), X_sub = X_sub, Y_sub = Y_sub) return model_sub, preds_sub def mnist_blackbox(train_start=0, train_end=60000, test_start=0, test_end=10000, nb_classes=10, batch_size=128, learning_rate=0.001, nb_epochs=10, holdout=150, data_aug=6, nb_epochs_s=10, lmbda=0.1, attack="fgsm", targeted=False): """ MNIST tutorial for the black-box attack from arxiv.org/abs/1602.02697 :param train_start: index of first training set example :param train_end: index of last training set example :param test_start: index of first test set example :param test_end: index of last test set example :return: a dictionary with: * black-box model accuracy on test set * substitute model accuracy on test set * black-box model accuracy on adversarial examples transferred from the substitute model """ keras.layers.core.K.set_learning_phase(0) # Dictionary used to keep track and return key accuracies accuracies = {} # Perform tutorial setup assert setup_tutorial() # Create TF session and set as Keras backend session gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.45) sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) keras.backend.set_session(sess) # Get MNIST data X_train, Y_train, X_test, Y_test = data_mnist(train_start=train_start, train_end=train_end, test_start=test_start, test_end=test_end) # Initialize substitute training set reserved for adversary X_sub = X_test[:holdout] Y_sub = np.argmax(Y_test[:holdout], axis=1) # Redefine test set as remaining samples unavailable to adversaries X_test = X_test[holdout:] Y_test = Y_test[holdout:] X_test = X_test[:FLAGS.n_attack] Y_test = Y_test[:FLAGS.n_attack] # Define input and output TF placeholders x = tf.placeholder(tf.float32, shape=(None, 28, 28, 1)) y = tf.placeholder(tf.float32, shape=(None, 10)) # Simulate the black-box model locally # You could replace this by a remote labeling API for instance print("Preparing the black-box model.") prep_bbox_out = prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate) model, bbox_preds, accuracies['bbox'] = prep_bbox_out # Train substitute using method from https://arxiv.org/abs/1602.02697 time_start = time.time() print("Training the substitute model.") train_sub_out = train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda) model_sub, preds_sub = train_sub_out time_end = time.time() print("Substitue model training time:", time_end - time_start) # Evaluate the substitute model on clean test examples eval_params = {'batch_size': batch_size} acc = model_eval(sess, x, y, preds_sub, X_test, Y_test, args=eval_params) accuracies['sub'] = acc print('substitution model accuracy:', acc) # Find the correctly predicted labels original_predict = batch_eval(sess, [x], [bbox_preds], [X_test], args=eval_params)[0] original_class = np.argmax(original_predict, axis = 1) true_class = np.argmax(Y_test, axis = 1) mask = true_class == original_class print(np.sum(mask), "out of", mask.size, "are correct labeled,", len(X_test[mask])) # Initialize the Fast Gradient Sign Method (FGSM) attack object. wrap = KerasModelWrapper(model_sub) # Craft adversarial examples using the substitute eval_params = {'batch_size': batch_size} if attack == "fgsm": attacker_params = {'eps': 0.4, 'ord': np.inf, 'clip_min': 0., 'clip_max': 1.} fgsm = FastGradientMethod(wrap, sess=sess) x_adv_sub = fgsm.generate(x, **attacker_params) attacker = fgsm adv_inputs = X_test ori_labels = Y_test print("Running FGSM attack...") else: print("Running Carlini and Wagner\'s L2 attack...") yname = "y" adv_ys = None # wrap = KerasModelWrapper(model) cwl2 = CarliniWagnerL2(wrap, back='tf', sess=sess) attacker_params = {'binary_search_steps': 9, 'max_iterations': 2000, 'abort_early': True, 'learning_rate': 0.01, 'batch_size': 1, 'initial_const': 0.01, 'confidence': 20} # generate targeted labels, 9 for each test example if targeted: adv_ys = [] targeted_class = [] for i in range(0, X_test.shape[0]): for j in range(0,10): # skip the original image label if j == np.argmax(Y_test[i]): continue adv_ys.append(np.eye(10)[j]) targeted_class.append(j) attacker_params['y_target'] = np.array(adv_ys, dtype=np.float32) # duplicate the inputs by 9 times adv_inputs = np.array([[instance] * 9 for instance in X_test], dtype=np.float32) adv_inputs = adv_inputs.reshape((X_test.shape[0] * 9, 28, 28, 1)) # also update the mask mask = np.repeat(mask, 9) ori_labels = np.repeat(Y_test, 9, axis=0) else: adv_inputs = X_test ori_labels = Y_test attacker = cwl2 if attack == "fgsm": # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, model(x_adv_sub), adv_inputs, ori_labels, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute: ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex'] = accuracy time_start = time.time() # Evaluate the accuracy of the "black-box" model on adversarial examples x_adv_sub_np = attacker.generate_np(adv_inputs, **attacker_params) accuracy = model_eval(sess, x, y, bbox_preds, x_adv_sub_np, ori_labels, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (NP): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex'] = accuracy time_end = time.time() print('Attack time:', time_end - time_start) # Evaluate the targeted attack bbox_adv_predict = batch_eval(sess, [x], [bbox_preds], [x_adv_sub_np], args=eval_params)[0] bbox_adv_class = np.argmax(bbox_adv_predict, axis = 1) true_class = np.argmax(ori_labels, axis = 1) untargeted_success = np.mean(bbox_adv_class != true_class) print('Untargeted attack success rate:', untargeted_success) accuracies['untargeted_success'] = untargeted_success if targeted: targeted_success = np.mean(bbox_adv_class == targeted_class) print('Targeted attack success rate:', targeted_success) accuracies['targeted_success'] = targeted_success if attack == "cwl2": # Compute the L2 pertubations of generated adversarial examples percent_perturbed = np.sum((x_adv_sub_np - adv_inputs)**2, axis=(1, 2, 3))**.5 # print(percent_perturbed) # print('Avg. L_2 norm of perturbations {0:.4f}'.format(np.mean(percent_perturbed))) # when computing the mean, removing the failure attacks first print('Avg. L_2 norm of perturbations {0:.4f}'.format(np.mean(percent_perturbed[percent_perturbed > 1e-8]))) # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, bbox_preds, adv_inputs[mask], ori_labels[mask], args=eval_params) print('Test accuracy of excluding originally incorrect labels (should be 1.0): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc_ori'] = accuracy if attack == "fgsm": # Evaluate the accuracy of the "black-box" model on adversarial examples (excluding correct) accuracy = model_eval(sess, x, y, model(x_adv_sub), adv_inputs[mask], ori_labels[mask], args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (excluding originally incorrect labels): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc'] = accuracy # Evaluate the accuracy of the "black-box" model on adversarial examples (excluding correct) x_adv_sub_mask_np = x_adv_sub_np[mask] accuracy = model_eval(sess, x, y, bbox_preds, x_adv_sub_mask_np, ori_labels[mask], args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (excluding originally incorrect labels, NP): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc'] = accuracy return accuracies def main(argv=None): mnist_blackbox(nb_classes=FLAGS.nb_classes, batch_size=FLAGS.batch_size, learning_rate=FLAGS.learning_rate, nb_epochs=FLAGS.nb_epochs, holdout=FLAGS.holdout, data_aug=FLAGS.data_aug, nb_epochs_s=FLAGS.nb_epochs_s, lmbda=FLAGS.lmbda, attack=FLAGS.attack, targeted=FLAGS.targeted) if __name__ == '__main__': # General flags flags.DEFINE_integer('nb_classes', 10, 'Number of classes in problem') flags.DEFINE_integer('batch_size', 128, 'Size of training batches') flags.DEFINE_integer('n_attack', -1, 'No. of images used for attack') flags.DEFINE_float('learning_rate', 0.001, 'Learning rate for training') # Flags related to oracle flags.DEFINE_integer('nb_epochs', 10, 'Number of epochs to train model') # Flags related to substitute flags.DEFINE_integer('holdout', 150, 'Test set holdout for adversary') flags.DEFINE_integer('data_aug', 6, 'Nb of substitute data augmentations') flags.DEFINE_integer('nb_epochs_s', 30, 'Training epochs for substitute') flags.DEFINE_float('lmbda', 0.1, 'Lambda from arxiv.org/abs/1602.02697') # Flags related to attack flags.DEFINE_string('attack', 'cwl2', 'cwl2 = Carlini & Wagner\'s L2 attack, fgsm = Fast Gradient Sign Method') flags.DEFINE_bool('targeted', False, 'use targeted attack') # Flags related to saving/loading flags.DEFINE_bool('load_pretrain', False, 'load pretrained model from sub_saved/mnist-model') flags.DEFINE_bool('cached_aug', False, 'use cached augmentation in sub_saved') flags.DEFINE_string('train_dir', 'sub_saved', 'model saving path') flags.DEFINE_string('filename', 'mnist-model', 'cifar model name') os.system("mkdir -p sub_saved") app.run() ================================================ FILE: retrain.py ================================================ # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== r"""Simple transfer learning with Inception v3 or Mobilenet models. With support for TensorBoard. This example shows how to take a Inception v3 or Mobilenet model trained on ImageNet images, and train a new top layer that can recognize other classes of images. The top layer receives as input a 2048-dimensional vector (1001-dimensional for Mobilenet) for each image. We train a softmax layer on top of this representation. Assuming the softmax layer contains N labels, this corresponds to learning N + 2048*N (or 1001*N) model parameters corresponding to the learned biases and weights. Here's an example, which assumes you have a folder containing class-named subfolders, each full of images for each label. The example folder flower_photos should have a structure like this: ~/flower_photos/daisy/photo1.jpg ~/flower_photos/daisy/photo2.jpg ... ~/flower_photos/rose/anotherphoto77.jpg ... ~/flower_photos/sunflower/somepicture.jpg The subfolder names are important, since they define what label is applied to each image, but the filenames themselves don't matter. Once your images are prepared, you can run the training with a command like this: ```bash bazel build tensorflow/examples/image_retraining:retrain && \ bazel-bin/tensorflow/examples/image_retraining/retrain \ --image_dir ~/flower_photos ``` Or, if you have a pip installation of tensorflow, `retrain.py` can be run without bazel: ```bash python tensorflow/examples/image_retraining/retrain.py \ --image_dir ~/flower_photos ``` You can replace the image_dir argument with any folder containing subfolders of images. The label for each image is taken from the name of the subfolder it's in. This produces a new model file that can be loaded and run by any TensorFlow program, for example the label_image sample code. By default this script will use the high accuracy, but comparatively large and slow Inception v3 model architecture. It's recommended that you start with this to validate that you have gathered good training data, but if you want to deploy on resource-limited platforms, you can try the `--architecture` flag with a Mobilenet model. For example: ```bash python tensorflow/examples/image_retraining/retrain.py \ --image_dir ~/flower_photos --architecture mobilenet_1.0_224 ``` There are 32 different Mobilenet models to choose from, with a variety of file size and latency options. The first number can be '1.0', '0.75', '0.50', or '0.25' to control the size, and the second controls the input image size, either '224', '192', '160', or '128', with smaller sizes running faster. See https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html for more information on Mobilenet. To use with TensorBoard: By default, this script will log summaries to /tmp/retrain_logs directory Visualize the summaries with this command: tensorboard --logdir /tmp/retrain_logs """ from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse from datetime import datetime import hashlib import os.path import random import re import sys import tarfile import numpy as np from six.moves import urllib import tensorflow as tf from tensorflow.python.framework import graph_util from tensorflow.python.framework import tensor_shape from tensorflow.python.platform import gfile from tensorflow.python.util import compat FLAGS = None # These are all parameters that are tied to the particular model architecture # we're using for Inception v3. These include things like tensor names and their # sizes. If you want to adapt this script to work with another model, you will # need to update these to reflect the values in the network you're using. MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1 # ~134M def create_image_lists(image_dir, testing_percentage, validation_percentage): """Builds a list of training images from the file system. Analyzes the sub folders in the image directory, splits them into stable training, testing, and validation sets, and returns a data structure describing the lists of images for each label and their paths. Args: image_dir: String path to a folder containing subfolders of images. testing_percentage: Integer percentage of the images to reserve for tests. validation_percentage: Integer percentage of images reserved for validation. Returns: A dictionary containing an entry for each label subfolder, with images split into training, testing, and validation sets within each label. """ if not gfile.Exists(image_dir): tf.logging.error("Image directory '" + image_dir + "' not found.") return None result = {} sub_dirs = [x[0] for x in gfile.Walk(image_dir)] # The root directory comes first, so skip it. is_root_dir = True for sub_dir in sub_dirs: if is_root_dir: is_root_dir = False continue extensions = ['jpg', 'jpeg', 'JPG', 'JPEG'] file_list = [] dir_name = os.path.basename(sub_dir) if dir_name == image_dir: continue tf.logging.info("Looking for images in '" + dir_name + "'") for extension in extensions: file_glob = os.path.join(image_dir, dir_name, '*.' + extension) file_list.extend(gfile.Glob(file_glob)) if not file_list: tf.logging.warning('No files found') continue if len(file_list) < 20: tf.logging.warning( 'WARNING: Folder has less than 20 images, which may cause issues.') elif len(file_list) > MAX_NUM_IMAGES_PER_CLASS: tf.logging.warning( 'WARNING: Folder {} has more than {} images. Some images will ' 'never be selected.'.format(dir_name, MAX_NUM_IMAGES_PER_CLASS)) label_name = re.sub(r'[^a-z0-9]+', ' ', dir_name.lower()) training_images = [] testing_images = [] validation_images = [] for file_name in file_list: base_name = os.path.basename(file_name) # We want to ignore anything after '_nohash_' in the file name when # deciding which set to put an image in, the data set creator has a way of # grouping photos that are close variations of each other. For example # this is used in the plant disease data set to group multiple pictures of # the same leaf. hash_name = re.sub(r'_nohash_.*$', '', file_name) # This looks a bit magical, but we need to decide whether this file should # go into the training, testing, or validation sets, and we want to keep # existing files in the same set even if more files are subsequently # added. # To do that, we need a stable way of deciding based on just the file name # itself, so we do a hash of that and then use that to generate a # probability value that we use to assign it. hash_name_hashed = hashlib.sha1(compat.as_bytes(hash_name)).hexdigest() percentage_hash = ((int(hash_name_hashed, 16) % (MAX_NUM_IMAGES_PER_CLASS + 1)) * (100.0 / MAX_NUM_IMAGES_PER_CLASS)) if percentage_hash < validation_percentage: validation_images.append(base_name) elif percentage_hash < (testing_percentage + validation_percentage): testing_images.append(base_name) else: training_images.append(base_name) result[label_name] = { 'dir': dir_name, 'training': training_images, 'testing': testing_images, 'validation': validation_images, } return result def get_image_path(image_lists, label_name, index, image_dir, category): """"Returns a path to an image for a label at the given index. Args: image_lists: Dictionary of training images for each label. label_name: Label string we want to get an image for. index: Int offset of the image we want. This will be moduloed by the available number of images for the label, so it can be arbitrarily large. image_dir: Root folder string of the subfolders containing the training images. category: Name string of set to pull images from - training, testing, or validation. Returns: File system path string to an image that meets the requested parameters. """ if label_name not in image_lists: tf.logging.fatal('Label does not exist %s.', label_name) label_lists = image_lists[label_name] if category not in label_lists: tf.logging.fatal('Category does not exist %s.', category) category_list = label_lists[category] if not category_list: tf.logging.fatal('Label %s has no images in the category %s.', label_name, category) mod_index = index % len(category_list) base_name = category_list[mod_index] sub_dir = label_lists['dir'] full_path = os.path.join(image_dir, sub_dir, base_name) return full_path def get_bottleneck_path(image_lists, label_name, index, bottleneck_dir, category, architecture): """"Returns a path to a bottleneck file for a label at the given index. Args: image_lists: Dictionary of training images for each label. label_name: Label string we want to get an image for. index: Integer offset of the image we want. This will be moduloed by the available number of images for the label, so it can be arbitrarily large. bottleneck_dir: Folder string holding cached files of bottleneck values. category: Name string of set to pull images from - training, testing, or validation. architecture: The name of the model architecture. Returns: File system path string to an image that meets the requested parameters. """ return get_image_path(image_lists, label_name, index, bottleneck_dir, category) + '_' + architecture + '.txt' def create_model_graph(model_info): """"Creates a graph from saved GraphDef file and returns a Graph object. Args: model_info: Dictionary containing information about the model architecture. Returns: Graph holding the trained Inception network, and various tensors we'll be manipulating. """ with tf.Graph().as_default() as graph: model_path = os.path.join(FLAGS.model_dir, model_info['model_file_name']) with gfile.FastGFile(model_path, 'rb') as f: graph_def = tf.GraphDef() graph_def.ParseFromString(f.read()) bottleneck_tensor, resized_input_tensor = (tf.import_graph_def( graph_def, name='', return_elements=[ model_info['bottleneck_tensor_name'], model_info['resized_input_tensor_name'], ])) return graph, bottleneck_tensor, resized_input_tensor def run_bottleneck_on_image(sess, image_data, image_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor): """Runs inference on an image to extract the 'bottleneck' summary layer. Args: sess: Current active TensorFlow Session. image_data: String of raw JPEG data. image_data_tensor: Input data layer in the graph. decoded_image_tensor: Output of initial image resizing and preprocessing. resized_input_tensor: The input node of the recognition graph. bottleneck_tensor: Layer before the final softmax. Returns: Numpy array of bottleneck values. """ # First decode the JPEG image, resize it, and rescale the pixel values. resized_input_values = sess.run(decoded_image_tensor, {image_data_tensor: image_data}) # Then run it through the recognition network. bottleneck_values = sess.run(bottleneck_tensor, {resized_input_tensor: resized_input_values}) bottleneck_values = np.squeeze(bottleneck_values) return bottleneck_values def maybe_download_and_extract(data_url): """Download and extract model tar file. If the pretrained model we're using doesn't already exist, this function downloads it from the TensorFlow.org website and unpacks it into a directory. Args: data_url: Web location of the tar file containing the pretrained model. """ dest_directory = FLAGS.model_dir if not os.path.exists(dest_directory): os.makedirs(dest_directory) filename = data_url.split('/')[-1] filepath = os.path.join(dest_directory, filename) if not os.path.exists(filepath): def _progress(count, block_size, total_size): sys.stdout.write('\r>> Downloading %s %.1f%%' % (filename, float(count * block_size) / float(total_size) * 100.0)) sys.stdout.flush() filepath, _ = urllib.request.urlretrieve(data_url, filepath, _progress) print() statinfo = os.stat(filepath) tf.logging.info('Successfully downloaded', filename, statinfo.st_size, 'bytes.') tarfile.open(filepath, 'r:gz').extractall(dest_directory) def ensure_dir_exists(dir_name): """Makes sure the folder exists on disk. Args: dir_name: Path string to the folder we want to create. """ if not os.path.exists(dir_name): os.makedirs(dir_name) bottleneck_path_2_bottleneck_values = {} def create_bottleneck_file(bottleneck_path, image_lists, label_name, index, image_dir, category, sess, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor): """Create a single bottleneck file.""" tf.logging.info('Creating bottleneck at ' + bottleneck_path) image_path = get_image_path(image_lists, label_name, index, image_dir, category) if not gfile.Exists(image_path): tf.logging.fatal('File does not exist %s', image_path) image_data = gfile.FastGFile(image_path, 'rb').read() try: bottleneck_values = run_bottleneck_on_image( sess, image_data, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor) except Exception as e: raise RuntimeError('Error during processing file %s (%s)' % (image_path, str(e))) bottleneck_string = ','.join(str(x) for x in bottleneck_values) with open(bottleneck_path, 'w') as bottleneck_file: bottleneck_file.write(bottleneck_string) def get_or_create_bottleneck(sess, image_lists, label_name, index, image_dir, category, bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture): """Retrieves or calculates bottleneck values for an image. If a cached version of the bottleneck data exists on-disk, return that, otherwise calculate the data and save it to disk for future use. Args: sess: The current active TensorFlow Session. image_lists: Dictionary of training images for each label. label_name: Label string we want to get an image for. index: Integer offset of the image we want. This will be modulo-ed by the available number of images for the label, so it can be arbitrarily large. image_dir: Root folder string of the subfolders containing the training images. category: Name string of which set to pull images from - training, testing, or validation. bottleneck_dir: Folder string holding cached files of bottleneck values. jpeg_data_tensor: The tensor to feed loaded jpeg data into. decoded_image_tensor: The output of decoding and resizing the image. resized_input_tensor: The input node of the recognition graph. bottleneck_tensor: The output tensor for the bottleneck values. architecture: The name of the model architecture. Returns: Numpy array of values produced by the bottleneck layer for the image. """ label_lists = image_lists[label_name] sub_dir = label_lists['dir'] sub_dir_path = os.path.join(bottleneck_dir, sub_dir) ensure_dir_exists(sub_dir_path) bottleneck_path = get_bottleneck_path(image_lists, label_name, index, bottleneck_dir, category, architecture) if not os.path.exists(bottleneck_path): create_bottleneck_file(bottleneck_path, image_lists, label_name, index, image_dir, category, sess, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor) with open(bottleneck_path, 'r') as bottleneck_file: bottleneck_string = bottleneck_file.read() did_hit_error = False try: bottleneck_values = [float(x) for x in bottleneck_string.split(',')] except ValueError: tf.logging.warning('Invalid float found, recreating bottleneck') did_hit_error = True if did_hit_error: create_bottleneck_file(bottleneck_path, image_lists, label_name, index, image_dir, category, sess, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor) with open(bottleneck_path, 'r') as bottleneck_file: bottleneck_string = bottleneck_file.read() # Allow exceptions to propagate here, since they shouldn't happen after a # fresh creation bottleneck_values = [float(x) for x in bottleneck_string.split(',')] return bottleneck_values def cache_bottlenecks(sess, image_lists, image_dir, bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture): """Ensures all the training, testing, and validation bottlenecks are cached. Because we're likely to read the same image multiple times (if there are no distortions applied during training) it can speed things up a lot if we calculate the bottleneck layer values once for each image during preprocessing, and then just read those cached values repeatedly during training. Here we go through all the images we've found, calculate those values, and save them off. Args: sess: The current active TensorFlow Session. image_lists: Dictionary of training images for each label. image_dir: Root folder string of the subfolders containing the training images. bottleneck_dir: Folder string holding cached files of bottleneck values. jpeg_data_tensor: Input tensor for jpeg data from file. decoded_image_tensor: The output of decoding and resizing the image. resized_input_tensor: The input node of the recognition graph. bottleneck_tensor: The penultimate output layer of the graph. architecture: The name of the model architecture. Returns: Nothing. """ how_many_bottlenecks = 0 ensure_dir_exists(bottleneck_dir) for label_name, label_lists in image_lists.items(): for category in ['training', 'testing', 'validation']: category_list = label_lists[category] for index, unused_base_name in enumerate(category_list): get_or_create_bottleneck( sess, image_lists, label_name, index, image_dir, category, bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture) how_many_bottlenecks += 1 if how_many_bottlenecks % 100 == 0: tf.logging.info( str(how_many_bottlenecks) + ' bottleneck files created.') def get_random_cached_bottlenecks(sess, image_lists, how_many, category, bottleneck_dir, image_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture): """Retrieves bottleneck values for cached images. If no distortions are being applied, this function can retrieve the cached bottleneck values directly from disk for images. It picks a random set of images from the specified category. Args: sess: Current TensorFlow Session. image_lists: Dictionary of training images for each label. how_many: If positive, a random sample of this size will be chosen. If negative, all bottlenecks will be retrieved. category: Name string of which set to pull from - training, testing, or validation. bottleneck_dir: Folder string holding cached files of bottleneck values. image_dir: Root folder string of the subfolders containing the training images. jpeg_data_tensor: The layer to feed jpeg image data into. decoded_image_tensor: The output of decoding and resizing the image. resized_input_tensor: The input node of the recognition graph. bottleneck_tensor: The bottleneck output layer of the CNN graph. architecture: The name of the model architecture. Returns: List of bottleneck arrays, their corresponding ground truths, and the relevant filenames. """ class_count = len(image_lists.keys()) bottlenecks = [] ground_truths = [] filenames = [] if how_many >= 0: # Retrieve a random sample of bottlenecks. for unused_i in range(how_many): label_index = random.randrange(class_count) label_name = list(image_lists.keys())[label_index] image_index = random.randrange(MAX_NUM_IMAGES_PER_CLASS + 1) image_name = get_image_path(image_lists, label_name, image_index, image_dir, category) bottleneck = get_or_create_bottleneck( sess, image_lists, label_name, image_index, image_dir, category, bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture) ground_truth = np.zeros(class_count, dtype=np.float32) ground_truth[label_index] = 1.0 bottlenecks.append(bottleneck) ground_truths.append(ground_truth) filenames.append(image_name) else: # Retrieve all bottlenecks. for label_index, label_name in enumerate(image_lists.keys()): for image_index, image_name in enumerate( image_lists[label_name][category]): image_name = get_image_path(image_lists, label_name, image_index, image_dir, category) bottleneck = get_or_create_bottleneck( sess, image_lists, label_name, image_index, image_dir, category, bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_input_tensor, bottleneck_tensor, architecture) ground_truth = np.zeros(class_count, dtype=np.float32) ground_truth[label_index] = 1.0 bottlenecks.append(bottleneck) ground_truths.append(ground_truth) filenames.append(image_name) return bottlenecks, ground_truths, filenames def get_random_distorted_bottlenecks( sess, image_lists, how_many, category, image_dir, input_jpeg_tensor, distorted_image, resized_input_tensor, bottleneck_tensor): """Retrieves bottleneck values for training images, after distortions. If we're training with distortions like crops, scales, or flips, we have to recalculate the full model for every image, and so we can't use cached bottleneck values. Instead we find random images for the requested category, run them through the distortion graph, and then the full graph to get the bottleneck results for each. Args: sess: Current TensorFlow Session. image_lists: Dictionary of training images for each label. how_many: The integer number of bottleneck values to return. category: Name string of which set of images to fetch - training, testing, or validation. image_dir: Root folder string of the subfolders containing the training images. input_jpeg_tensor: The input layer we feed the image data to. distorted_image: The output node of the distortion graph. resized_input_tensor: The input node of the recognition graph. bottleneck_tensor: The bottleneck output layer of the CNN graph. Returns: List of bottleneck arrays and their corresponding ground truths. """ class_count = len(image_lists.keys()) bottlenecks = [] ground_truths = [] for unused_i in range(how_many): label_index = random.randrange(class_count) label_name = list(image_lists.keys())[label_index] image_index = random.randrange(MAX_NUM_IMAGES_PER_CLASS + 1) image_path = get_image_path(image_lists, label_name, image_index, image_dir, category) if not gfile.Exists(image_path): tf.logging.fatal('File does not exist %s', image_path) jpeg_data = gfile.FastGFile(image_path, 'rb').read() # Note that we materialize the distorted_image_data as a numpy array before # sending running inference on the image. This involves 2 memory copies and # might be optimized in other implementations. distorted_image_data = sess.run(distorted_image, {input_jpeg_tensor: jpeg_data}) bottleneck_values = sess.run(bottleneck_tensor, {resized_input_tensor: distorted_image_data}) bottleneck_values = np.squeeze(bottleneck_values) ground_truth = np.zeros(class_count, dtype=np.float32) ground_truth[label_index] = 1.0 bottlenecks.append(bottleneck_values) ground_truths.append(ground_truth) return bottlenecks, ground_truths def should_distort_images(flip_left_right, random_crop, random_scale, random_brightness): """Whether any distortions are enabled, from the input flags. Args: flip_left_right: Boolean whether to randomly mirror images horizontally. random_crop: Integer percentage setting the total margin used around the crop box. random_scale: Integer percentage of how much to vary the scale by. random_brightness: Integer range to randomly multiply the pixel values by. Returns: Boolean value indicating whether any distortions should be applied. """ return (flip_left_right or (random_crop != 0) or (random_scale != 0) or (random_brightness != 0)) def add_input_distortions(flip_left_right, random_crop, random_scale, random_brightness, input_width, input_height, input_depth, input_mean, input_std): """Creates the operations to apply the specified distortions. During training it can help to improve the results if we run the images through simple distortions like crops, scales, and flips. These reflect the kind of variations we expect in the real world, and so can help train the model to cope with natural data more effectively. Here we take the supplied parameters and construct a network of operations to apply them to an image. Cropping ~~~~~~~~ Cropping is done by placing a bounding box at a random position in the full image. The cropping parameter controls the size of that box relative to the input image. If it's zero, then the box is the same size as the input and no cropping is performed. If the value is 50%, then the crop box will be half the width and height of the input. In a diagram it looks like this: < width > +---------------------+ | | | width - crop% | | < > | | +------+ | | | | | | | | | | | | | | +------+ | | | | | +---------------------+ Scaling ~~~~~~~ Scaling is a lot like cropping, except that the bounding box is always centered and its size varies randomly within the given range. For example if the scale percentage is zero, then the bounding box is the same size as the input and no scaling is applied. If it's 50%, then the bounding box will be in a random range between half the width and height and full size. Args: flip_left_right: Boolean whether to randomly mirror images horizontally. random_crop: Integer percentage setting the total margin used around the crop box. random_scale: Integer percentage of how much to vary the scale by. random_brightness: Integer range to randomly multiply the pixel values by. graph. input_width: Horizontal size of expected input image to model. input_height: Vertical size of expected input image to model. input_depth: How many channels the expected input image should have. input_mean: Pixel value that should be zero in the image for the graph. input_std: How much to divide the pixel values by before recognition. Returns: The jpeg input layer and the distorted result tensor. """ jpeg_data = tf.placeholder(tf.string, name='DistortJPGInput') decoded_image = tf.image.decode_jpeg(jpeg_data, channels=input_depth) decoded_image_as_float = tf.cast(decoded_image, dtype=tf.float32) decoded_image_4d = tf.expand_dims(decoded_image_as_float, 0) margin_scale = 1.0 + (random_crop / 100.0) resize_scale = 1.0 + (random_scale / 100.0) margin_scale_value = tf.constant(margin_scale) resize_scale_value = tf.random_uniform(tensor_shape.scalar(), minval=1.0, maxval=resize_scale) scale_value = tf.multiply(margin_scale_value, resize_scale_value) precrop_width = tf.multiply(scale_value, input_width) precrop_height = tf.multiply(scale_value, input_height) precrop_shape = tf.stack([precrop_height, precrop_width]) precrop_shape_as_int = tf.cast(precrop_shape, dtype=tf.int32) precropped_image = tf.image.resize_bilinear(decoded_image_4d, precrop_shape_as_int) precropped_image_3d = tf.squeeze(precropped_image, squeeze_dims=[0]) cropped_image = tf.random_crop(precropped_image_3d, [input_height, input_width, input_depth]) if flip_left_right: flipped_image = tf.image.random_flip_left_right(cropped_image) else: flipped_image = cropped_image brightness_min = 1.0 - (random_brightness / 100.0) brightness_max = 1.0 + (random_brightness / 100.0) brightness_value = tf.random_uniform(tensor_shape.scalar(), minval=brightness_min, maxval=brightness_max) brightened_image = tf.multiply(flipped_image, brightness_value) offset_image = tf.subtract(brightened_image, input_mean) mul_image = tf.multiply(offset_image, 1.0 / input_std) distort_result = tf.expand_dims(mul_image, 0, name='DistortResult') return jpeg_data, distort_result def variable_summaries(var): """Attach a lot of summaries to a Tensor (for TensorBoard visualization).""" with tf.name_scope('summaries'): mean = tf.reduce_mean(var) tf.summary.scalar('mean', mean) with tf.name_scope('stddev'): stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean))) tf.summary.scalar('stddev', stddev) tf.summary.scalar('max', tf.reduce_max(var)) tf.summary.scalar('min', tf.reduce_min(var)) tf.summary.histogram('histogram', var) def add_final_training_ops(class_count, final_tensor_name, bottleneck_tensor, bottleneck_tensor_size): """Adds a new softmax and fully-connected layer for training. We need to retrain the top layer to identify our new classes, so this function adds the right operations to the graph, along with some variables to hold the weights, and then sets up all the gradients for the backward pass. The set up for the softmax and fully-connected layers is based on: https://www.tensorflow.org/versions/master/tutorials/mnist/beginners/index.html Args: class_count: Integer of how many categories of things we're trying to recognize. final_tensor_name: Name string for the new final node that produces results. bottleneck_tensor: The output of the main CNN graph. bottleneck_tensor_size: How many entries in the bottleneck vector. Returns: The tensors for the training and cross entropy results, and tensors for the bottleneck input and ground truth input. """ with tf.name_scope('input'): bottleneck_input = tf.placeholder_with_default( bottleneck_tensor, shape=[None, bottleneck_tensor_size], name='BottleneckInputPlaceholder') ground_truth_input = tf.placeholder(tf.float32, [None, class_count], name='GroundTruthInput') # Organizing the following ops as `final_training_ops` so they're easier # to see in TensorBoard layer_name = 'final_training_ops' with tf.name_scope(layer_name): with tf.name_scope('weights'): initial_value = tf.truncated_normal( [bottleneck_tensor_size, class_count], stddev=0.001) layer_weights = tf.Variable(initial_value, name='final_weights') variable_summaries(layer_weights) with tf.name_scope('biases'): layer_biases = tf.Variable(tf.zeros([class_count]), name='final_biases') variable_summaries(layer_biases) with tf.name_scope('Wx_plus_b'): logits = tf.matmul(bottleneck_input, layer_weights) + layer_biases tf.summary.histogram('pre_activations', logits) final_tensor = tf.nn.softmax(logits, name=final_tensor_name) tf.summary.histogram('activations', final_tensor) with tf.name_scope('cross_entropy'): cross_entropy = tf.nn.softmax_cross_entropy_with_logits( labels=ground_truth_input, logits=logits) with tf.name_scope('total'): cross_entropy_mean = tf.reduce_mean(cross_entropy) tf.summary.scalar('cross_entropy', cross_entropy_mean) with tf.name_scope('train'): optimizer = tf.train.GradientDescentOptimizer(FLAGS.learning_rate) train_step = optimizer.minimize(cross_entropy_mean) return (train_step, cross_entropy_mean, bottleneck_input, ground_truth_input, final_tensor) def add_evaluation_step(result_tensor, ground_truth_tensor): """Inserts the operations we need to evaluate the accuracy of our results. Args: result_tensor: The new final node that produces results. ground_truth_tensor: The node we feed ground truth data into. Returns: Tuple of (evaluation step, prediction). """ with tf.name_scope('accuracy'): with tf.name_scope('correct_prediction'): prediction = tf.argmax(result_tensor, 1) correct_prediction = tf.equal( prediction, tf.argmax(ground_truth_tensor, 1)) with tf.name_scope('accuracy'): evaluation_step = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) tf.summary.scalar('accuracy', evaluation_step) return evaluation_step, prediction def save_graph_to_file(sess, graph, graph_file_name): output_graph_def = graph_util.convert_variables_to_constants( sess, graph.as_graph_def(), [FLAGS.final_tensor_name]) with gfile.FastGFile(graph_file_name, 'wb') as f: f.write(output_graph_def.SerializeToString()) return def prepare_file_system(): # Setup the directory we'll write summaries to for TensorBoard if tf.gfile.Exists(FLAGS.summaries_dir): tf.gfile.DeleteRecursively(FLAGS.summaries_dir) tf.gfile.MakeDirs(FLAGS.summaries_dir) if FLAGS.intermediate_store_frequency > 0: ensure_dir_exists(FLAGS.intermediate_output_graphs_dir) return def create_model_info(architecture): """Given the name of a model architecture, returns information about it. There are different base image recognition pretrained models that can be retrained using transfer learning, and this function translates from the name of a model to the attributes that are needed to download and train with it. Args: architecture: Name of a model architecture. Returns: Dictionary of information about the model, or None if the name isn't recognized Raises: ValueError: If architecture name is unknown. """ architecture = architecture.lower() if architecture == 'inception_v3': # pylint: disable=line-too-long data_url = 'http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz' # pylint: enable=line-too-long bottleneck_tensor_name = 'pool_3/_reshape:0' bottleneck_tensor_size = 2048 input_width = 299 input_height = 299 input_depth = 3 resized_input_tensor_name = 'Mul:0' model_file_name = 'classify_image_graph_def.pb' input_mean = 128 input_std = 128 elif architecture.startswith('mobilenet_'): parts = architecture.split('_') if len(parts) != 3 and len(parts) != 4: tf.logging.error("Couldn't understand architecture name '%s'", architecture) return None version_string = parts[1] if (version_string != '1.0' and version_string != '0.75' and version_string != '0.50' and version_string != '0.25'): tf.logging.error( """"The Mobilenet version should be '1.0', '0.75', '0.50', or '0.25', but found '%s' for architecture '%s'""", version_string, architecture) return None size_string = parts[2] if (size_string != '224' and size_string != '192' and size_string != '160' and size_string != '128'): tf.logging.error( """The Mobilenet input size should be '224', '192', '160', or '128', but found '%s' for architecture '%s'""", size_string, architecture) return None if len(parts) == 3: is_quantized = False else: if parts[3] != 'quantized': tf.logging.error( "Couldn't understand architecture suffix '%s' for '%s'", parts[3], architecture) return None is_quantized = True data_url = 'http://download.tensorflow.org/models/mobilenet_v1_' data_url += version_string + '_' + size_string + '_frozen.tgz' bottleneck_tensor_name = 'MobilenetV1/Predictions/Reshape:0' bottleneck_tensor_size = 1001 input_width = int(size_string) input_height = int(size_string) input_depth = 3 resized_input_tensor_name = 'input:0' if is_quantized: model_base_name = 'quantized_graph.pb' else: model_base_name = 'frozen_graph.pb' model_dir_name = 'mobilenet_v1_' + version_string + '_' + size_string model_file_name = os.path.join(model_dir_name, model_base_name) input_mean = 127.5 input_std = 127.5 else: tf.logging.error("Couldn't understand architecture name '%s'", architecture) raise ValueError('Unknown architecture', architecture) return { 'data_url': data_url, 'bottleneck_tensor_name': bottleneck_tensor_name, 'bottleneck_tensor_size': bottleneck_tensor_size, 'input_width': input_width, 'input_height': input_height, 'input_depth': input_depth, 'resized_input_tensor_name': resized_input_tensor_name, 'model_file_name': model_file_name, 'input_mean': input_mean, 'input_std': input_std, } def add_jpeg_decoding(input_width, input_height, input_depth, input_mean, input_std): """Adds operations that perform JPEG decoding and resizing to the graph.. Args: input_width: Desired width of the image fed into the recognizer graph. input_height: Desired width of the image fed into the recognizer graph. input_depth: Desired channels of the image fed into the recognizer graph. input_mean: Pixel value that should be zero in the image for the graph. input_std: How much to divide the pixel values by before recognition. Returns: Tensors for the node to feed JPEG data into, and the output of the preprocessing steps. """ jpeg_data = tf.placeholder(tf.string, name='DecodeJPGInput') decoded_image = tf.image.decode_jpeg(jpeg_data, channels=input_depth) decoded_image_as_float = tf.cast(decoded_image, dtype=tf.float32) decoded_image_4d = tf.expand_dims(decoded_image_as_float, 0) resize_shape = tf.stack([input_height, input_width]) resize_shape_as_int = tf.cast(resize_shape, dtype=tf.int32) resized_image = tf.image.resize_bilinear(decoded_image_4d, resize_shape_as_int) offset_image = tf.subtract(resized_image, input_mean) mul_image = tf.multiply(offset_image, 1.0 / input_std) return jpeg_data, mul_image def main(_): # Needed to make sure the logging output is visible. # See https://github.com/tensorflow/tensorflow/issues/3047 tf.logging.set_verbosity(tf.logging.INFO) # Prepare necessary directories that can be used during training prepare_file_system() # Gather information about the model architecture we'll be using. model_info = create_model_info(FLAGS.architecture) if not model_info: tf.logging.error('Did not recognize architecture flag') return -1 # Set up the pre-trained graph. maybe_download_and_extract(model_info['data_url']) graph, bottleneck_tensor, resized_image_tensor = ( create_model_graph(model_info)) # Look at the folder structure, and create lists of all the images. image_lists = create_image_lists(FLAGS.image_dir, FLAGS.testing_percentage, FLAGS.validation_percentage) class_count = len(image_lists.keys()) if class_count == 0: tf.logging.error('No valid folders of images found at ' + FLAGS.image_dir) return -1 if class_count == 1: tf.logging.error('Only one valid folder of images found at ' + FLAGS.image_dir + ' - multiple classes are needed for classification.') return -1 # See if the command-line flags mean we're applying any distortions. do_distort_images = should_distort_images( FLAGS.flip_left_right, FLAGS.random_crop, FLAGS.random_scale, FLAGS.random_brightness) with tf.Session(graph=graph) as sess: # Set up the image decoding sub-graph. jpeg_data_tensor, decoded_image_tensor = add_jpeg_decoding( model_info['input_width'], model_info['input_height'], model_info['input_depth'], model_info['input_mean'], model_info['input_std']) if do_distort_images: # We will be applying distortions, so setup the operations we'll need. (distorted_jpeg_data_tensor, distorted_image_tensor) = add_input_distortions( FLAGS.flip_left_right, FLAGS.random_crop, FLAGS.random_scale, FLAGS.random_brightness, model_info['input_width'], model_info['input_height'], model_info['input_depth'], model_info['input_mean'], model_info['input_std']) else: # We'll make sure we've calculated the 'bottleneck' image summaries and # cached them on disk. cache_bottlenecks(sess, image_lists, FLAGS.image_dir, FLAGS.bottleneck_dir, jpeg_data_tensor, decoded_image_tensor, resized_image_tensor, bottleneck_tensor, FLAGS.architecture) # Add the new layer that we'll be training. (train_step, cross_entropy, bottleneck_input, ground_truth_input, final_tensor) = add_final_training_ops( len(image_lists.keys()), FLAGS.final_tensor_name, bottleneck_tensor, model_info['bottleneck_tensor_size']) # Create the operations we need to evaluate the accuracy of our new layer. evaluation_step, prediction = add_evaluation_step( final_tensor, ground_truth_input) # Merge all the summaries and write them out to the summaries_dir merged = tf.summary.merge_all() train_writer = tf.summary.FileWriter(FLAGS.summaries_dir + '/train', sess.graph) validation_writer = tf.summary.FileWriter( FLAGS.summaries_dir + '/validation') # Set up all our weights to their initial default values. init = tf.global_variables_initializer() sess.run(init) # Run the training for as many cycles as requested on the command line. for i in range(FLAGS.how_many_training_steps): # Get a batch of input bottleneck values, either calculated fresh every # time with distortions applied, or from the cache stored on disk. if do_distort_images: (train_bottlenecks, train_ground_truth) = get_random_distorted_bottlenecks( sess, image_lists, FLAGS.train_batch_size, 'training', FLAGS.image_dir, distorted_jpeg_data_tensor, distorted_image_tensor, resized_image_tensor, bottleneck_tensor) else: (train_bottlenecks, train_ground_truth, _) = get_random_cached_bottlenecks( sess, image_lists, FLAGS.train_batch_size, 'training', FLAGS.bottleneck_dir, FLAGS.image_dir, jpeg_data_tensor, decoded_image_tensor, resized_image_tensor, bottleneck_tensor, FLAGS.architecture) # Feed the bottlenecks and ground truth into the graph, and run a training # step. Capture training summaries for TensorBoard with the `merged` op. train_summary, _ = sess.run( [merged, train_step], feed_dict={bottleneck_input: train_bottlenecks, ground_truth_input: train_ground_truth}) train_writer.add_summary(train_summary, i) # Every so often, print out how well the graph is training. is_last_step = (i + 1 == FLAGS.how_many_training_steps) if (i % FLAGS.eval_step_interval) == 0 or is_last_step: train_accuracy, cross_entropy_value = sess.run( [evaluation_step, cross_entropy], feed_dict={bottleneck_input: train_bottlenecks, ground_truth_input: train_ground_truth}) tf.logging.info('%s: Step %d: Train accuracy = %.1f%%' % (datetime.now(), i, train_accuracy * 100)) tf.logging.info('%s: Step %d: Cross entropy = %f' % (datetime.now(), i, cross_entropy_value)) validation_bottlenecks, validation_ground_truth, _ = ( get_random_cached_bottlenecks( sess, image_lists, FLAGS.validation_batch_size, 'validation', FLAGS.bottleneck_dir, FLAGS.image_dir, jpeg_data_tensor, decoded_image_tensor, resized_image_tensor, bottleneck_tensor, FLAGS.architecture)) # Run a validation step and capture training summaries for TensorBoard # with the `merged` op. validation_summary, validation_accuracy = sess.run( [merged, evaluation_step], feed_dict={bottleneck_input: validation_bottlenecks, ground_truth_input: validation_ground_truth}) validation_writer.add_summary(validation_summary, i) tf.logging.info('%s: Step %d: Validation accuracy = %.1f%% (N=%d)' % (datetime.now(), i, validation_accuracy * 100, len(validation_bottlenecks))) # Store intermediate results intermediate_frequency = FLAGS.intermediate_store_frequency if (intermediate_frequency > 0 and (i % intermediate_frequency == 0) and i > 0): intermediate_file_name = (FLAGS.intermediate_output_graphs_dir + 'intermediate_' + str(i) + '.pb') tf.logging.info('Save intermediate result to : ' + intermediate_file_name) save_graph_to_file(sess, graph, intermediate_file_name) # We've completed all our training, so run a final test evaluation on # some new images we haven't used before. test_bottlenecks, test_ground_truth, test_filenames = ( get_random_cached_bottlenecks( sess, image_lists, FLAGS.test_batch_size, 'testing', FLAGS.bottleneck_dir, FLAGS.image_dir, jpeg_data_tensor, decoded_image_tensor, resized_image_tensor, bottleneck_tensor, FLAGS.architecture)) test_accuracy, predictions = sess.run( [evaluation_step, prediction], feed_dict={bottleneck_input: test_bottlenecks, ground_truth_input: test_ground_truth}) tf.logging.info('Final test accuracy = %.1f%% (N=%d)' % (test_accuracy * 100, len(test_bottlenecks))) if FLAGS.print_misclassified_test_images: tf.logging.info('=== MISCLASSIFIED TEST IMAGES ===') for i, test_filename in enumerate(test_filenames): if predictions[i] != test_ground_truth[i].argmax(): tf.logging.info('%70s %s' % (test_filename, list(image_lists.keys())[predictions[i]])) # Write out the trained graph and labels with the weights stored as # constants. save_graph_to_file(sess, graph, FLAGS.output_graph) with gfile.FastGFile(FLAGS.output_labels, 'w') as f: f.write('\n'.join(image_lists.keys()) + '\n') if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( '--image_dir', type=str, default='', help='Path to folders of labeled images.' ) parser.add_argument( '--output_graph', type=str, default='/tmp/output_graph.pb', help='Where to save the trained graph.' ) parser.add_argument( '--intermediate_output_graphs_dir', type=str, default='/tmp/intermediate_graph/', help='Where to save the intermediate graphs.' ) parser.add_argument( '--intermediate_store_frequency', type=int, default=0, help="""\ How many steps to store intermediate graph. If "0" then will not store.\ """ ) parser.add_argument( '--output_labels', type=str, default='/tmp/output_labels.txt', help='Where to save the trained graph\'s labels.' ) parser.add_argument( '--summaries_dir', type=str, default='/tmp/retrain_logs', help='Where to save summary logs for TensorBoard.' ) parser.add_argument( '--how_many_training_steps', type=int, default=4000, help='How many training steps to run before ending.' ) parser.add_argument( '--learning_rate', type=float, default=0.01, help='How large a learning rate to use when training.' ) parser.add_argument( '--testing_percentage', type=int, default=10, help='What percentage of images to use as a test set.' ) parser.add_argument( '--validation_percentage', type=int, default=10, help='What percentage of images to use as a validation set.' ) parser.add_argument( '--eval_step_interval', type=int, default=10, help='How often to evaluate the training results.' ) parser.add_argument( '--train_batch_size', type=int, default=100, help='How many images to train on at a time.' ) parser.add_argument( '--test_batch_size', type=int, default=-1, help="""\ How many images to test on. This test set is only used once, to evaluate the final accuracy of the model after training completes. A value of -1 causes the entire test set to be used, which leads to more stable results across runs.\ """ ) parser.add_argument( '--validation_batch_size', type=int, default=100, help="""\ How many images to use in an evaluation batch. This validation set is used much more often than the test set, and is an early indicator of how accurate the model is during training. A value of -1 causes the entire validation set to be used, which leads to more stable results across training iterations, but may be slower on large training sets.\ """ ) parser.add_argument( '--print_misclassified_test_images', default=False, help="""\ Whether to print out a list of all misclassified test images.\ """, action='store_true' ) parser.add_argument( '--model_dir', type=str, default='/tmp/imagenet', help="""\ Path to classify_image_graph_def.pb, imagenet_synset_to_human_label_map.txt, and imagenet_2012_challenge_label_map_proto.pbtxt.\ """ ) parser.add_argument( '--bottleneck_dir', type=str, default='/tmp/bottleneck', help='Path to cache bottleneck layer values as files.' ) parser.add_argument( '--final_tensor_name', type=str, default='final_result', help="""\ The name of the output classification layer in the retrained graph.\ """ ) parser.add_argument( '--flip_left_right', default=False, help="""\ Whether to randomly flip half of the training images horizontally.\ """, action='store_true' ) parser.add_argument( '--random_crop', type=int, default=0, help="""\ A percentage determining how much of a margin to randomly crop off the training images.\ """ ) parser.add_argument( '--random_scale', type=int, default=0, help="""\ A percentage determining how much to randomly scale up the size of the training images by.\ """ ) parser.add_argument( '--random_brightness', type=int, default=0, help="""\ A percentage determining how much to randomly multiply the training image input pixels up or down by.\ """ ) parser.add_argument( '--architecture', type=str, default='inception_v3', help="""\ Which model architecture to use. 'inception_v3' is the most accurate, but also the slowest. For faster or smaller models, chose a MobileNet with the form 'mobilenet__[_quantized]'. For example, 'mobilenet_1.0_224' will pick a model that is 17 MB in size and takes 224 pixel input images, while 'mobilenet_0.25_128_quantized' will choose a much less accurate, but smaller and faster network that's 920 KB on disk and takes 128x128 images. See https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html for more information on Mobilenet.\ """) FLAGS, unparsed = parser.parse_known_args() tf.app.run(main=main, argv=[sys.argv[0]] + unparsed) ================================================ FILE: setup_cifar.py ================================================ ## setup_cifar.py -- cifar data and model loading code ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import tensorflow as tf import numpy as np import os import pickle import gzip import pickle import urllib.request from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Conv2D, MaxPooling2D from keras.utils import np_utils from keras.models import load_model def load_batch(fpath, label_key='labels'): f = open(fpath, 'rb') d = pickle.load(f, encoding="bytes") for k, v in d.items(): del(d[k]) d[k.decode("utf8")] = v f.close() data = d["data"] labels = d[label_key] data = data.reshape(data.shape[0], 3, 32, 32) final = np.zeros((data.shape[0], 32, 32, 3),dtype=np.float32) final[:,:,:,0] = data[:,0,:,:] final[:,:,:,1] = data[:,1,:,:] final[:,:,:,2] = data[:,2,:,:] final /= 255 final -= .5 labels2 = np.zeros((len(labels), 10)) labels2[np.arange(len(labels2)), labels] = 1 return final, labels def load_batch(fpath): f = open(fpath,"rb").read() size = 32*32*3+1 labels = [] images = [] for i in range(10000): arr = np.fromstring(f[i*size:(i+1)*size],dtype=np.uint8) lab = np.identity(10)[arr[0]] img = arr[1:].reshape((3,32,32)).transpose((1,2,0)) labels.append(lab) images.append((img/255)-.5) return np.array(images),np.array(labels) class CIFAR: def __init__(self): train_data = [] train_labels = [] if not os.path.exists("cifar-10-batches-bin"): urllib.request.urlretrieve("https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz", "cifar-data.tar.gz") os.popen("tar -xzf cifar-data.tar.gz").read() for i in range(5): r,s = load_batch("cifar-10-batches-bin/data_batch_"+str(i+1)+".bin") train_data.extend(r) train_labels.extend(s) train_data = np.array(train_data,dtype=np.float32) train_labels = np.array(train_labels) self.test_data, self.test_labels = load_batch("cifar-10-batches-bin/test_batch.bin") VALIDATION_SIZE = 5000 self.validation_data = train_data[:VALIDATION_SIZE, :, :, :] self.validation_labels = train_labels[:VALIDATION_SIZE] self.train_data = train_data[VALIDATION_SIZE:, :, :, :] self.train_labels = train_labels[VALIDATION_SIZE:] class CIFARModel: def __init__(self, restore=None, session=None, use_log=False): self.num_channels = 3 self.image_size = 32 self.num_labels = 10 model = Sequential() model.add(Conv2D(64, (3, 3), input_shape=(32, 32, 3))) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Conv2D(128, (3, 3))) model.add(Activation('relu')) model.add(Conv2D(128, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Flatten()) model.add(Dense(256)) model.add(Activation('relu')) model.add(Dense(256)) model.add(Activation('relu')) model.add(Dense(10)) if use_log: model.add(Activation('softmax')) if restore: model.load_weights(restore) self.model = model def predict(self, data): return self.model(data) ================================================ FILE: setup_inception.py ================================================ ## Copyright (C) IBM Corp, 2017-2018 ## Modified by Huan Zhang for the updated Inception-v3 model (inception_v3_2016_08_28.tar.gz) ## Modified by Nicholas Carlini to match model structure for attack code. ## Original copyright license follows. # Copyright 2015 The TensorFlow Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== """Simple image classification with Inception. Run image classification with Inception trained on ImageNet 2012 Challenge data set. This program creates a graph from a saved GraphDef protocol buffer, and runs inference on an input JPEG image. It outputs human readable strings of the top 5 predictions along with their probabilities. Change the --image_file argument to any jpg image to compute a classification of that image. Please see the tutorial and website for a detailed description of how to use this script to perform image recognition. https://tensorflow.org/tutorials/image_recognition/ """ from __future__ import absolute_import from __future__ import division from __future__ import print_function import os.path import re import sys import random import tarfile import scipy.misc import numpy as np from six.moves import urllib import tensorflow as tf # pylint: disable=line-too-long DATA_URL = 'http://download.huan-zhang.com/models/adv/imagenet/inception_v3_2016_08_28_frozen.tar.gz' # pylint: enable=line-too-long class NodeLookup(object): """Converts integer node ID's to human readable labels.""" def __init__(self, label_lookup_path=None): if not label_lookup_path: label_lookup_path = os.path.join( FLAGS.model_dir, 'labels.txt') self.node_lookup = self.load(label_lookup_path) def load(self, label_lookup_path): """Loads a human readable English name for each softmax node. Args: label_lookup_path: string UID to integer node ID. uid_lookup_path: string UID to human-readable string. Returns: dict from integer node ID to human-readable string. """ if not tf.gfile.Exists(label_lookup_path): tf.logging.fatal('File does not exist %s', label_lookup_path) # Loads mapping from string UID to integer node ID. node_id_to_name = {} proto_as_ascii = tf.gfile.GFile(label_lookup_path).readlines() for line in proto_as_ascii: if line: words = line.split(':') target_class = int(words[0]) name = words[1] node_id_to_name[target_class] = name return node_id_to_name def id_to_string(self, node_id): if node_id not in self.node_lookup: return '' return self.node_lookup[node_id] def create_graph(): """Creates a graph from saved GraphDef file and returns a saver.""" # Creates graph from saved graph_def.pb. with tf.gfile.FastGFile(os.path.join( # FLAGS.model_dir, 'classify_image_graph_def.pb'), 'rb') as f: FLAGS.model_dir, 'frozen_inception_v3.pb'), 'rb') as f: graph_def = tf.GraphDef() graph_def.ParseFromString(f.read()) #for line in repr(graph_def).split("\n"): # if "tensor_content" not in line: # print(line) _ = tf.import_graph_def(graph_def, name='') def run_inference_on_image(image): """Runs inference on an image. (Not updated, not working for inception v3 20160828) Args: image: Image file name. Returns: Nothing """ if not tf.gfile.Exists(image): tf.logging.fatal('File does not exist %s', image) image_data = tf.gfile.FastGFile(image, 'rb').read() # Creates graph from saved GraphDef. create_graph() with tf.Session() as sess: # Some useful tensors: # 'softmax:0': A tensor containing the normalized prediction across # 1000 labels. # 'pool_3:0': A tensor containing the next-to-last layer containing 2048 # float description of the image. # 'DecodeJpeg/contents:0': A tensor containing a string providing JPEG # encoding of the image. # Runs the softmax tensor by feeding the image_data as input to the graph. #softmax_tensor = sess.graph.get_tensor_by_name('softmax:0') img = tf.placeholder(tf.uint8, (299,299,3)) softmax_tensor = tf.import_graph_def( sess.graph.as_graph_def(), input_map={'DecodeJpeg:0': tf.reshape(img,((299,299,3)))}, return_elements=['softmax/logits:0']) dat = scipy.misc.imresize(scipy.misc.imread(image),(299,299)) predictions = sess.run(softmax_tensor, {img: dat}) predictions = np.squeeze(predictions) # Creates node ID --> English string lookup. node_lookup = NodeLookup() top_k = predictions.argsort()#[-FLAGS.num_top_predictions:][::-1] for node_id in top_k: print('id',node_id) human_string = node_lookup.id_to_string(node_id) score = predictions[node_id] print('%s (score = %.5f)' % (human_string, score)) class InceptionModelPrediction: def __init__(self, sess, use_log = False): self.sess = sess self.use_log = use_log if self.use_log: output_name = 'InceptionV3/Predictions/Softmax:0' else: output_name = 'InceptionV3/Predictions/Reshape:0' self.img = tf.placeholder(tf.float32, (None, 299,299,3)) self.softmax_tensor = tf.import_graph_def( sess.graph.as_graph_def(), input_map={'input:0': self.img}, return_elements=[output_name]) def predict(self, dat): dat = np.squeeze(dat) # scaled = (0.5 + dat) * 255 scaled = dat.reshape((1,) + dat.shape) # print(scaled.shape) predictions = self.sess.run(self.softmax_tensor, {self.img: scaled}) predictions = np.squeeze(predictions) return predictions # Creates node ID --> English string lookup. node_lookup = NodeLookup() top_k = predictions.argsort()#[-FLAGS.num_top_predictions:][::-1] for node_id in top_k: print('id',node_id) human_string = node_lookup.id_to_string(node_id) score = predictions[node_id] print('%s (score = %.5f)' % (human_string, score)) return top_k[-1] CREATED_GRAPH = False class InceptionModel: image_size = 299 num_labels = 1001 num_channels = 3 def __init__(self, sess, use_log = False): global CREATED_GRAPH self.sess = sess self.use_log = use_log if not CREATED_GRAPH: create_graph() CREATED_GRAPH = True self.model = InceptionModelPrediction(sess, use_log) def predict(self, img): if self.use_log: output_name = 'InceptionV3/Predictions/Softmax:0' else: output_name = 'InceptionV3/Predictions/Reshape:0' # scaled = (0.5+tf.reshape(img,((299,299,3))))*255 # scaled = (0.5+img)*255 if img.shape.as_list()[0]: # check if a shape has been specified explicitly shape = (int(img.shape[0]), 1001) softmax_tensor = tf.import_graph_def( self.sess.graph.as_graph_def(), input_map={'input:0': img, 'InceptionV3/Predictions/Shape:0': shape}, return_elements=[output_name]) else: # placeholder shape softmax_tensor = tf.import_graph_def( self.sess.graph.as_graph_def(), input_map={'input:0': img}, return_elements=[output_name]) return softmax_tensor[0] def maybe_download_and_extract(): """Download and extract model tar file.""" dest_directory = FLAGS.model_dir if not os.path.exists(dest_directory): os.makedirs(dest_directory) filename = DATA_URL.split('/')[-1] filepath = os.path.join(dest_directory, filename) if not os.path.exists(filepath): def _progress(count, block_size, total_size): sys.stdout.write('\r>> Downloading %s %.1f%%' % ( filename, float(count * block_size) / float(total_size) * 100.0)) sys.stdout.flush() filepath, _ = urllib.request.urlretrieve(DATA_URL, filepath, _progress) print() statinfo = os.stat(filepath) print('Succesfully downloaded', filename, statinfo.st_size, 'bytes.') tarfile.open(filepath, 'r:gz').extractall(dest_directory) def main(_): maybe_download_and_extract() image = (FLAGS.image_file if FLAGS.image_file else os.path.join(FLAGS.model_dir, 'cropped_panda.jpg')) # run_inference_on_image(image) create_graph() with tf.Session() as sess: dat = np.array(scipy.misc.imresize(scipy.misc.imread(image),(299,299)), dtype = np.float32) dat /= 255.0 dat -= 0.5 # print(dat) model = InceptionModelPrediction(sess, True) predictions = model.predict(dat) # Creates node ID --> English string lookup. node_lookup = NodeLookup() top_k = predictions.argsort()#[-FLAGS.num_top_predictions:][::-1] for node_id in top_k: print('id',node_id) human_string = node_lookup.id_to_string(node_id) score = predictions[node_id] print('%s (score = %.5f)' % (human_string, score)) def readimg(ff): f = "../imagenetdata/imgs/"+ff img = scipy.misc.imread(f) # skip small images (image should be at least 299x299) if img.shape[0] < 299 or img.shape[1] < 299: return None img = np.array(scipy.misc.imresize(img,(299,299)),dtype=np.float32)/255-.5 if img.shape != (299, 299, 3): return None return [img, int(ff.split(".")[0])] class ImageNet: def __init__(self): from multiprocessing import Pool pool = Pool(8) file_list = sorted(os.listdir("../imagenetdata/imgs/")) random.shuffle(file_list) r = pool.map(readimg, file_list[:200]) print(file_list[:200]) r = [x for x in r if x != None] test_data, test_labels = zip(*r) self.test_data = np.array(test_data) self.test_labels = np.zeros((len(test_labels), 1001)) self.test_labels[np.arange(len(test_labels)), test_labels] = 1 if __name__ == '__main__': FLAGS = tf.app.flags.FLAGS # classify_image_graph_def.pb: # Binary representation of the GraphDef protocol buffer. # imagenet_synset_to_human_label_map.txt: # Map from synset ID to a human readable string. # imagenet_2012_challenge_label_map_proto.pbtxt: # Text representation of a protocol buffer mapping a label to synset ID. tf.app.flags.DEFINE_string( 'model_dir', 'tmp/imagenet', """Path to classify_image_graph_def.pb, """ """imagenet_synset_to_human_label_map.txt, and """ """imagenet_2012_challenge_label_map_proto.pbtxt.""") tf.app.flags.DEFINE_string('image_file', '', """Absolute path to image file.""") tf.app.flags.DEFINE_integer('num_top_predictions', 5, """Display this many predictions.""") tf.app.run() else: from argparse import Namespace FLAGS = Namespace(model_dir="tmp/imagenet") ================================================ FILE: setup_mnist.py ================================================ ## setup_mnist.py -- mnist data and model loading code ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import tensorflow as tf import numpy as np import os import pickle import gzip import urllib.request from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Conv2D, MaxPooling2D from keras.utils import np_utils from keras.models import load_model def extract_data(filename, num_images): with gzip.open(filename) as bytestream: bytestream.read(16) buf = bytestream.read(num_images*28*28) data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32) data = (data / 255) - 0.5 data = data.reshape(num_images, 28, 28, 1) return data def extract_labels(filename, num_images): with gzip.open(filename) as bytestream: bytestream.read(8) buf = bytestream.read(1 * num_images) labels = np.frombuffer(buf, dtype=np.uint8) return (np.arange(10) == labels[:, None]).astype(np.float32) class MNIST: def __init__(self): if not os.path.exists("data"): os.mkdir("data") files = ["train-images-idx3-ubyte.gz", "t10k-images-idx3-ubyte.gz", "train-labels-idx1-ubyte.gz", "t10k-labels-idx1-ubyte.gz"] for name in files: urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/' + name, "data/"+name) train_data = extract_data("data/train-images-idx3-ubyte.gz", 60000) train_labels = extract_labels("data/train-labels-idx1-ubyte.gz", 60000) self.test_data = extract_data("data/t10k-images-idx3-ubyte.gz", 10000) self.test_labels = extract_labels("data/t10k-labels-idx1-ubyte.gz", 10000) VALIDATION_SIZE = 5000 self.validation_data = train_data[:VALIDATION_SIZE, :, :, :] self.validation_labels = train_labels[:VALIDATION_SIZE] self.train_data = train_data[VALIDATION_SIZE:, :, :, :] self.train_labels = train_labels[VALIDATION_SIZE:] class MNISTModel: def __init__(self, restore = None, session=None, use_log=False): self.num_channels = 1 self.image_size = 28 self.num_labels = 10 model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=(28, 28, 1))) model.add(Activation('relu')) model.add(Conv2D(32, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Flatten()) model.add(Dense(200)) model.add(Activation('relu')) model.add(Dense(200)) model.add(Activation('relu')) model.add(Dense(10)) # output log probability, used for black-box attack if use_log: model.add(Activation('softmax')) if restore: model.load_weights(restore) self.model = model def predict(self, data): return self.model(data) ================================================ FILE: substitute_blackbox.py ================================================ ## Copyright (C) IBM Corp, 2017-2018 from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals import os import time import numpy as np from six.moves import xrange import keras from keras import backend from keras.utils.np_utils import to_categorical from keras.models import Sequential from keras.layers import Dense, Flatten, Activation, Dropout from keras.datasets import cifar10 from keras.utils import np_utils import tensorflow as tf from tensorflow.python.platform import app from tensorflow.python.platform import flags from cleverhans.utils_keras import cnn_model from cleverhans.utils_mnist import data_mnist from cleverhans.utils_tf import model_train, model_eval, batch_eval, tf_model_load from cleverhans.attacks import CarliniWagnerL2 from cleverhans.attacks import FastGradientMethod from cleverhans.attacks_tf import jacobian_graph, jacobian_augmentation from cleverhans.utils_keras import KerasModelWrapper from setup_mnist import MNISTModel from setup_cifar import CIFARModel FLAGS = flags.FLAGS DATASET = "cifar" def data_cifar10(): """ Preprocess CIFAR10 dataset :return: """ # These values are specific to CIFAR10 img_rows = 32 img_cols = 32 nb_classes = 10 # the data, shuffled and split between train and test sets (X_train, y_train), (X_test, y_test) = cifar10.load_data() if keras.backend.image_dim_ordering() == 'th': X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols) X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols) else: X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3) X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3) X_train = X_train.astype('float32') X_test = X_test.astype('float32') X_train /= 255 X_test /= 255 print('X_train shape:', X_train.shape) print(X_train.shape[0], 'train samples') print(X_test.shape[0], 'test samples') # convert class vectors to binary class matrices Y_train = np_utils.to_categorical(y_train, nb_classes) Y_test = np_utils.to_categorical(y_test, nb_classes) return X_train, Y_train, X_test, Y_test def setup_tutorial(): """ Helper function to check correct configuration of tf and keras for tutorial :return: True if setup checks completed """ # Set TF random seed to improve reproducibility tf.set_random_seed(1234) if not hasattr(backend, "tf"): raise RuntimeError("This tutorial requires keras to be configured" " to use the TensorFlow backend.") # Image dimensions ordering should follow the Theano convention if keras.backend.image_dim_ordering() != 'tf': keras.backend.set_image_dim_ordering('tf') print("INFO: '~/.keras/keras.json' sets 'image_dim_ordering' " "to 'th', temporarily setting to 'tf'") return True def prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate): """ Define and train a model that simulates the "remote" black-box oracle described in the original paper. :param sess: the TF session :param x: the input placeholder for MNIST :param y: the ouput placeholder for MNIST :param X_train: the training data for the oracle :param Y_train: the training labels for the oracle :param X_test: the testing data for the oracle :param Y_test: the testing labels for the oracle :param nb_epochs: number of epochs to train model :param batch_size: size of training batches :param learning_rate: learning rate for training :return: """ # Define TF model graph (for the black-box model) if DATASET == "mnist": model = MNISTModel(use_log = True).model else: model = CIFARModel(use_log = True).model predictions = model(x) print("Defined TensorFlow model graph.") # Train an MNIST model if FLAGS.load_pretrain: tf_model_load(sess) else: train_params = { 'nb_epochs': nb_epochs, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, predictions, X_train, Y_train, verbose=True, save=True, args=train_params) # Print out the accuracy on legitimate data eval_params = {'batch_size': batch_size} accuracy = model_eval(sess, x, y, predictions, X_test, Y_test, args=eval_params) print('Test accuracy of black-box on legitimate test ' 'examples: ' + str(accuracy)) return model, predictions, accuracy def substitute_model(img_rows=28, img_cols=28, nb_classes=10): """ Defines the model architecture to be used by the substitute :param img_rows: number of rows in input :param img_cols: number of columns in input :param nb_classes: number of classes in output :return: keras model """ model = Sequential() # Find out the input shape ordering if keras.backend.image_dim_ordering() == 'th': input_shape = (1, img_rows, img_cols) else: input_shape = (img_rows, img_cols, 1) # Define a fully connected model (it's different than the black-box) layers = [Flatten(input_shape=input_shape), Dense(200), Activation('relu'), Dropout(0.5), Dense(200), Activation('relu'), Dropout(0.5), Dense(nb_classes), Activation('softmax')] for layer in layers: model.add(layer) return model def train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda): """ This function creates the substitute by alternatively augmenting the training data and training the substitute. :param sess: TF session :param x: input TF placeholder :param y: output TF placeholder :param bbox_preds: output of black-box model predictions :param X_sub: initial substitute training data :param Y_sub: initial substitute training labels :param nb_classes: number of output classes :param nb_epochs_s: number of epochs to train substitute model :param batch_size: size of training batches :param learning_rate: learning rate for training :param data_aug: number of times substitute training data is augmented :param lmbda: lambda from arxiv.org/abs/1602.02697 :return: """ # Define TF model graph (for the black-box model) # model_sub = substitute_model() if DATASET == "mnist": model_sub = MNISTModel(use_log = True).model else: model_sub = CIFARModel(use_log = True).model preds_sub = model_sub(x) print("Defined TensorFlow model graph for the substitute.") # Define the Jacobian symbolically using TensorFlow grads = jacobian_graph(preds_sub, x, nb_classes) # Train the substitute and augment dataset alternatively for rho in xrange(data_aug): print("Substitute training epoch #" + str(rho)) train_params = { 'nb_epochs': nb_epochs_s, 'batch_size': batch_size, 'learning_rate': learning_rate } model_train(sess, x, y, preds_sub, X_sub, to_categorical(Y_sub), init_all=False, verbose=False, args=train_params) # If we are not at last substitute training iteration, augment dataset if rho < data_aug - 1: if FLAGS.cached_aug: augs = np.load('sub_saved/{}-aug-{}.npz'.format(DATASET, rho)) X_sub = augs['X_sub'] Y_sub = augs['Y_sub'] else: print("Augmenting substitute training data.") # Perform the Jacobian augmentation X_sub = jacobian_augmentation(sess, x, X_sub, Y_sub, grads, lmbda) print("Labeling substitute training data.") # Label the newly generated synthetic points using the black-box Y_sub = np.hstack([Y_sub, Y_sub]) X_sub_prev = X_sub[int(len(X_sub)/2):] eval_params = {'batch_size': batch_size} bbox_val = batch_eval(sess, [x], [bbox_preds], [X_sub_prev], args=eval_params)[0] # Note here that we take the argmax because the adversary # only has access to the label (not the probabilities) output # by the black-box model Y_sub[int(len(X_sub)/2):] = np.argmax(bbox_val, axis=1) # cache the augmentation if not FLAGS.cached_aug: np.savez('sub_saved/{}-aug-{}.npz'.format(DATASET, rho), X_sub = X_sub, Y_sub = Y_sub) return model_sub, preds_sub def mnist_blackbox(train_start=0, train_end=60000, test_start=0, test_end=10000, nb_classes=10, batch_size=128, learning_rate=0.001, nb_epochs=10, holdout=150, data_aug=6, nb_epochs_s=10, lmbda=0.1, attack="fgsm", targeted=False): """ MNIST tutorial for the black-box attack from arxiv.org/abs/1602.02697 :param train_start: index of first training set example :param train_end: index of last training set example :param test_start: index of first test set example :param test_end: index of last test set example :return: a dictionary with: * black-box model accuracy on test set * substitute model accuracy on test set * black-box model accuracy on adversarial examples transferred from the substitute model """ keras.layers.core.K.set_learning_phase(0) # Dictionary used to keep track and return key accuracies accuracies = {} # Perform tutorial setup assert setup_tutorial() # Create TF session and set as Keras backend session gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=1.0) sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) keras.backend.set_session(sess) # Get MNIST data if DATASET == "mnist": X_train, Y_train, X_test, Y_test = data_mnist(train_start=train_start, train_end=train_end, test_start=test_start, test_end=test_end) else: X_train, Y_train, X_test, Y_test = data_cifar10() # Initialize substitute training set reserved for adversary X_sub = X_test[:holdout] Y_sub = np.argmax(Y_test[:holdout], axis=1) # Redefine test set as remaining samples unavailable to adversaries X_test = X_test[holdout:] Y_test = Y_test[holdout:] X_test = X_test[:FLAGS.n_attack] Y_test = Y_test[:FLAGS.n_attack] # Define input and output TF placeholders if DATASET == "mnist": x = tf.placeholder(tf.float32, shape=(None, 28, 28, 1)) else: x = tf.placeholder(tf.float32, shape=(None, 32, 32, 3)) y = tf.placeholder(tf.float32, shape=(None, 10)) # for feed targeted attack labels t_y = tf.placeholder(tf.float32, shape=(None, 10)) # Simulate the black-box model locally # You could replace this by a remote labeling API for instance print("Preparing the black-box model.") prep_bbox_out = prep_bbox(sess, x, y, X_train, Y_train, X_test, Y_test, nb_epochs, batch_size, learning_rate) model, bbox_preds, accuracies['bbox'] = prep_bbox_out # Train substitute using method from https://arxiv.org/abs/1602.02697 time_start = time.time() print("Training the substitute model.") train_sub_out = train_sub(sess, x, y, bbox_preds, X_sub, Y_sub, nb_classes, nb_epochs_s, batch_size, learning_rate, data_aug, lmbda) model_sub, preds_sub = train_sub_out time_end = time.time() print("Substitue model training time:", time_end - time_start) # Evaluate the substitute model on clean test examples eval_params = {'batch_size': batch_size} acc = model_eval(sess, x, y, preds_sub, X_test, Y_test, args=eval_params) accuracies['sub'] = acc print('substitution model accuracy:', acc) # Find the correctly predicted labels original_predict = batch_eval(sess, [x], [bbox_preds], [X_test], args=eval_params)[0] original_class = np.argmax(original_predict, axis = 1) true_class = np.argmax(Y_test, axis = 1) mask = true_class == original_class print(np.sum(mask), "out of", mask.size, "are correct labeled,", len(X_test[mask])) # Initialize the Fast Gradient Sign Method (FGSM) attack object. wrap = KerasModelWrapper(model_sub) # Craft adversarial examples using the substitute if targeted and attack == "fgsm": # TODO: fix the batch size mess eval_params = {'batch_size': FLAGS.n_attack * 9} else: eval_params = {'batch_size': batch_size} adv_inputs = X_test ori_labels = Y_test # generate targeted labels, 9 for each test example if targeted: adv_ys = [] targeted_class = [] for i in range(0, X_test.shape[0]): for j in range(0,10): # skip the original image label if j == np.argmax(Y_test[i]): continue adv_ys.append(np.eye(10)[j]) targeted_class.append(j) # duplicate the inputs by 9 times adv_inputs = np.array([[instance] * 9 for instance in X_test], dtype=np.float32) if DATASET == "mnist": adv_inputs = adv_inputs.reshape((X_test.shape[0] * 9, 28, 28, 1)) else: adv_inputs = adv_inputs.reshape((X_test.shape[0] * 9, 32, 32, 3)) # also update the mask mask = np.repeat(mask, 9) ori_labels = np.repeat(Y_test, 9, axis=0) adv_ys = np.array(adv_ys, dtype=np.float32) if attack == "fgsm": attacker_params = {'eps': 0.4, 'ord': np.inf, 'clip_min': 0., 'clip_max': 1.} # wrap = KerasModelWrapper(model) fgsm = FastGradientMethod(wrap, sess=sess) attacker = fgsm print("Running FGSM attack...") if targeted: attacker_params['y_target'] = t_y x_adv_sub = fgsm.generate(x, **attacker_params) else: print("Running Carlini and Wagner\'s L2 attack...") yname = "y" adv_ys = None # wrap = KerasModelWrapper(model) cwl2 = CarliniWagnerL2(wrap, back='tf', sess=sess) attacker_params = {'binary_search_steps': 9, 'max_iterations': 2000, 'abort_early': True, 'learning_rate': 0.01, 'batch_size': 1, 'initial_const': 0.01, 'confidence': 20} # generate targeted labels, 9 for each test example if targeted: attacker_params['y_target'] = adv_ys # attacker_params['batch_size'] = 9 attacker = cwl2 time_start = time.time() if attack == "fgsm": # Evaluate the accuracy of the "black-box" model on adversarial examples if targeted: accuracy = model_eval(sess, x, y, model(x_adv_sub), adv_inputs, ori_labels, feed={t_y: adv_ys}, args=eval_params) else: accuracy = model_eval(sess, x, y, model(x_adv_sub), adv_inputs, ori_labels, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute: ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex'] = accuracy else: # Evaluate the accuracy of the "black-box" model on adversarial examples x_adv_sub_np = attacker.generate_np(adv_inputs, **attacker_params) accuracy = model_eval(sess, x, y, bbox_preds, x_adv_sub_np, ori_labels, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (NP): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex'] = accuracy time_end = time.time() print('Attack time:', time_end - time_start) # Evaluate the targeted attack if attack == "fgsm": bbox_adv_predict = batch_eval(sess, [x], [model(x_adv_sub)], [adv_inputs], feed={t_y: adv_ys}, args=eval_params)[0] else: bbox_adv_predict = batch_eval(sess, [x], [bbox_preds], [x_adv_sub_np], args=eval_params)[0] bbox_adv_class = np.argmax(bbox_adv_predict, axis = 1) print(bbox_adv_class) print(true_class) true_class = np.argmax(ori_labels, axis = 1) untargeted_success = np.mean(bbox_adv_class != true_class) print('Untargeted attack success rate:', untargeted_success) accuracies['untargeted_success'] = untargeted_success if targeted: targeted_success = np.mean(bbox_adv_class == targeted_class) print('Targeted attack success rate:', targeted_success) accuracies['targeted_success'] = targeted_success if attack == "cwl2": # Compute the L2 pertubations of generated adversarial examples percent_perturbed = np.sum((x_adv_sub_np - adv_inputs)**2, axis=(1, 2, 3))**.5 print(percent_perturbed) # print('Avg. L_2 norm of perturbations {0:.4f}'.format(np.mean(percent_perturbed))) # when computing the mean, removing the failure attacks first print('Avg. L_2 norm of all perturbations {0:.4f}'.format(np.mean(percent_perturbed[percent_perturbed > 1e-8]))) print('Avg. L_2 norm of successful untargeted perturbations {0:.4f}'.format(np.mean(percent_perturbed[bbox_adv_class != true_class]))) if targeted: print('Avg. L_2 norm of successful targeted perturbations {0:.4f}'.format(np.mean(percent_perturbed[bbox_adv_class == targeted_class]))) # Evaluate the accuracy of the "black-box" model on adversarial examples accuracy = model_eval(sess, x, y, bbox_preds, adv_inputs[mask], ori_labels[mask], args=eval_params) print('Test accuracy of excluding originally incorrect labels (should be 1.0): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc_ori'] = accuracy if attack == "fgsm": # Evaluate the accuracy of the "black-box" model on adversarial examples (excluding correct) accuracy = model_eval(sess, x, y, model(x_adv_sub), adv_inputs[mask], ori_labels[mask], feed={t_y: adv_ys[mask]}, args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (excluding originally incorrect labels): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc'] = accuracy else: # Evaluate the accuracy of the "black-box" model on adversarial examples (excluding correct) x_adv_sub_mask_np = x_adv_sub_np[mask] accuracy = model_eval(sess, x, y, bbox_preds, x_adv_sub_mask_np, ori_labels[mask], args=eval_params) print('Test accuracy of oracle on adversarial examples generated ' 'using the substitute (excluding originally incorrect labels, NP): ' + str(accuracy)) accuracies['bbox_on_sub_adv_ex_exc'] = accuracy return accuracies def main(argv=None): print("DATASET:", DATASET) print("Targeted:", FLAGS.targeted) print("Attack:", FLAGS.attack) print("Use Pretrained", FLAGS.load_pretrain) print("Train Epochs:", FLAGS.nb_epochs) print("Sub Train Epochs:", FLAGS.nb_epochs_s) print("Holdout Size:", FLAGS.holdout) print("Data Augmentation:", FLAGS.data_aug) print("Number of Attacks:", FLAGS.n_attack) mnist_blackbox(nb_classes=FLAGS.nb_classes, batch_size=FLAGS.batch_size, learning_rate=FLAGS.learning_rate, nb_epochs=FLAGS.nb_epochs, holdout=FLAGS.holdout, data_aug=FLAGS.data_aug, nb_epochs_s=FLAGS.nb_epochs_s, lmbda=FLAGS.lmbda, attack=FLAGS.attack, targeted=FLAGS.targeted) if __name__ == '__main__': # General flags flags.DEFINE_integer('nb_classes', 10, 'Number of classes in problem') flags.DEFINE_integer('batch_size', 128, 'Size of training batches') flags.DEFINE_integer('n_attack', -1, 'No. of images used for attack') if DATASET == "mnist": flags.DEFINE_float('learning_rate', 0.001, 'Learning rate for training') else: flags.DEFINE_float('learning_rate', 0.0005, 'Learning rate for training') # Flags related to oracle if DATASET == "mnist": flags.DEFINE_integer('nb_epochs', 10, 'Number of epochs to train model') else: flags.DEFINE_integer('nb_epochs', 50, 'Number of epochs to train model') # Flags related to substitute flags.DEFINE_integer('holdout', 150, 'Test set holdout for adversary') flags.DEFINE_integer('data_aug', 6, 'Nb of substitute data augmentations') if DATASET == "mnist": flags.DEFINE_integer('nb_epochs_s', 30, 'Training epochs for substitute') else: flags.DEFINE_integer('nb_epochs_s', 50, 'Training epochs for substitute') flags.DEFINE_float('lmbda', 0.1, 'Lambda from arxiv.org/abs/1602.02697') # Flags related to attack flags.DEFINE_string('attack', 'cwl2', 'cwl2 = Carlini & Wagner\'s L2 attack, fgsm = Fast Gradient Sign Method') flags.DEFINE_bool('targeted', False, 'use targeted attack') # Flags related to saving/loading flags.DEFINE_bool('load_pretrain', False, 'load pretrained model from sub_saved/mnist-model') flags.DEFINE_bool('cached_aug', False, 'use cached augmentation in sub_saved') flags.DEFINE_string('train_dir', 'sub_saved', 'model saving path') if DATASET == "mnist": flags.DEFINE_string('filename', 'mnist-model', 'mnist model name') else: flags.DEFINE_string('filename', 'cifar-model', 'cifar model name') os.system("mkdir -p sub_saved") app.run() ================================================ FILE: test_all.py ================================================ ## test_attack.py -- sample code to test attack procedure ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2017, Huan Zhang . ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import os import sys import tensorflow as tf import numpy as np import random import time from setup_cifar import CIFAR, CIFARModel from setup_mnist import MNIST, MNISTModel from setup_inception import ImageNet, InceptionModel from l2_attack import CarliniL2 from l2_attack_black import BlackBoxL2 from PIL import Image def show(img, name = "output.png"): """ Show MNSIT digits in the console. """ np.save(name, img) fig = np.around((img + 0.5)*255) fig = fig.astype(np.uint8).squeeze() pic = Image.fromarray(fig) # pic.resize((512,512), resample=PIL.Image.BICUBIC) pic.save(name) remap = " .*#"+"#"*100 img = (img.flatten()+.5)*3 if len(img) != 784: return print("START") for i in range(28): print("".join([remap[int(round(x))] for x in img[i*28:i*28+28]])) def generate_data(data, samples, targeted=True, start=0, inception=False): """ Generate the input data to the attack algorithm. data: the images to attack samples: number of samples to use targeted: if true, construct targeted attacks, otherwise untargeted attacks start: offset into data to use inception: if targeted and inception, randomly sample 100 targets intead of 1000 """ inputs = [] targets = [] labels = [] true_ids = [] for i in range(samples): if targeted: if inception: # for inception, randomly choose 10 target classes seq = np.random.choice(range(1,1001), 10) # seq = [580] # grand piano else: # for CIFAR and MNIST, generate all target classes seq = range(data.test_labels.shape[1]) # print ('image label:', np.argmax(data.test_labels[start+i])) for j in seq: # skip the original image label if (j == np.argmax(data.test_labels[start+i])) and (inception == False): continue inputs.append(data.test_data[start+i]) targets.append(np.eye(data.test_labels.shape[1])[j]) labels.append(data.test_labels[start+i]) true_ids.append(start+i) else: inputs.append(data.test_data[start+i]) targets.append(data.test_labels[start+i]) labels.append(data.test_labels[start+i]) true_ids.append(start+i) inputs = np.array(inputs) targets = np.array(targets) labels = np.array(labels) true_ids = np.array(true_ids) return inputs, targets, labels, true_ids def main(args): with tf.Session() as sess: use_log = not args['use_zvalue'] is_inception = args['dataset'] == "imagenet" # load network print('Loading model', args['dataset']) if args['dataset'] == "mnist": data, model = MNIST(), MNISTModel("models/mnist", sess, use_log) # data, model = MNIST(), MNISTModel("models/mnist-distilled-100", sess, use_log) elif args['dataset'] == "cifar10": data, model = CIFAR(), CIFARModel("models/cifar", sess, use_log) # data, model = CIFAR(), CIFARModel("models/cifar-distilled-100", sess, use_log) elif args['dataset'] == "imagenet": data, model = ImageNet(), InceptionModel(sess, use_log) print('Done...') if args['numimg'] == 0: args['numimg'] = len(data.test_labels) - args['firstimg'] print('Using', args['numimg'], 'test images') # load attack module if args['attack'] == "white": # batch size 1, optimize on 1 image at a time, rather than optimizing images jointly attack = CarliniL2(sess, model, batch_size=1, max_iterations=args['maxiter'], print_every=args['print_every'], early_stop_iters=args['early_stop_iters'], confidence=0, learning_rate = args['lr'], initial_const=args['init_const'], binary_search_steps=args['binary_steps'], targeted=not args['untargeted'], use_log=use_log, adam_beta1=args['adam_beta1'], adam_beta2=args['adam_beta2']) else: # batch size 128, optimize on 128 coordinates of a single image attack = BlackBoxL2(sess, model, batch_size=128, max_iterations=args['maxiter'], print_every=args['print_every'], early_stop_iters=args['early_stop_iters'], confidence=0, learning_rate = args['lr'], initial_const=args['init_const'], binary_search_steps=args['binary_steps'], targeted=not args['untargeted'], use_log=use_log, use_tanh=args['use_tanh'], use_resize=args['use_resize'], adam_beta1=args['adam_beta1'], adam_beta2=args['adam_beta2'], reset_adam_after_found=args['reset_adam'], solver=args['solver'], save_ckpts=args['save_ckpts'], load_checkpoint=args['load_ckpt'], start_iter=args['start_iter'], init_size=args['init_size'], use_importance=not args['uniform']) random.seed(args['seed']) np.random.seed(args['seed']) print('Generate data') all_inputs, all_targets, all_labels, all_true_ids = generate_data(data, samples=args['numimg'], targeted=not args['untargeted'], start=args['firstimg'], inception=is_inception) print('Done...') os.system("mkdir -p {}/{}".format(args['save'], args['dataset'])) img_no = 0 total_success = 0 l2_total = 0.0 for i in range(all_true_ids.size): inputs = all_inputs[i:i+1] targets = all_targets[i:i+1] labels = all_labels[i:i+1] print("true labels:", np.argmax(labels), labels) print("target:", np.argmax(targets), targets) # test if the image is correctly classified original_predict = model.model.predict(inputs) original_predict = np.squeeze(original_predict) original_prob = np.sort(original_predict) original_class = np.argsort(original_predict) print("original probabilities:", original_prob[-1:-6:-1]) print("original classification:", original_class[-1:-6:-1]) print("original probabilities (most unlikely):", original_prob[:6]) print("original classification (most unlikely):", original_class[:6]) if original_class[-1] != np.argmax(labels): print("skip wrongly classified image no. {}, original class {}, classified as {}".format(i, np.argmax(labels), original_class[-1])) continue img_no += 1 timestart = time.time() adv, const = attack.attack_batch(inputs, targets) if type(const) is list: const = const[0] if len(adv.shape) == 3: adv = adv.reshape((1,) + adv.shape) timeend = time.time() l2_distortion = np.sum((adv-inputs)**2)**.5 adversarial_predict = model.model.predict(adv) adversarial_predict = np.squeeze(adversarial_predict) adversarial_prob = np.sort(adversarial_predict) adversarial_class = np.argsort(adversarial_predict) print("adversarial probabilities:", adversarial_prob[-1:-6:-1]) print("adversarial classification:", adversarial_class[-1:-6:-1]) success = False if args['untargeted']: if adversarial_class[-1] != original_class[-1]: success = True else: if adversarial_class[-1] == np.argmax(targets): success = True if l2_distortion > 20.0: success = False if success: total_success += 1 l2_total += l2_distortion suffix = "id{}_seq{}_prev{}_adv{}_{}_dist{}".format(all_true_ids[i], i, original_class[-1], adversarial_class[-1], success, l2_distortion) print("Saving to", suffix) show(inputs, "{}/{}/{}_original_{}.png".format(args['save'], args['dataset'], img_no, suffix)) show(adv, "{}/{}/{}_adversarial_{}.png".format(args['save'], args['dataset'], img_no, suffix)) show(adv - inputs, "{}/{}/{}_diff_{}.png".format(args['save'], args['dataset'], img_no, suffix)) print("[STATS][L1] total = {}, seq = {}, id = {}, time = {:.3f}, success = {}, const = {:.6f}, prev_class = {}, new_class = {}, distortion = {:.5f}, success_rate = {:.3f}, l2_avg = {:.5f}".format(img_no, i, all_true_ids[i], timeend - timestart, success, const, original_class[-1], adversarial_class[-1], l2_distortion, total_success / float(img_no), 0 if total_success == 0 else l2_total / total_success)) sys.stdout.flush() # t = np.random.randn(28*28).reshape(1,28,28,1) # print(model.model.predict(t)) if __name__ == "__main__": import argparse parser = argparse.ArgumentParser() parser.add_argument("-d", "--dataset", choices=["mnist", "cifar10", "imagenet"], default="mnist") parser.add_argument("-s", "--save", default="./saved_results") parser.add_argument("-a", "--attack", choices=["white", "black"], default="white") parser.add_argument("-n", "--numimg", type=int, default=0, help = "number of test images to attack") parser.add_argument("-m", "--maxiter", type=int, default=0, help = "set 0 to use default value") parser.add_argument("-p", "--print_every", type=int, default=100, help = "print objs every PRINT_EVERY iterations") parser.add_argument("-o", "--early_stop_iters", type=int, default=100, help = "print objs every EARLY_STOP_ITER iterations, 0 is maxiter//10") parser.add_argument("-f", "--firstimg", type=int, default=0) parser.add_argument("-b", "--binary_steps", type=int, default=0) parser.add_argument("-c", "--init_const", type=float, default=0.0) parser.add_argument("-z", "--use_zvalue", action='store_true') parser.add_argument("-u", "--untargeted", action='store_true') parser.add_argument("-r", "--reset_adam", action='store_true', help = "reset adam after an initial solution is found") parser.add_argument("--use_resize", action='store_true', help = "resize image (only works on imagenet!)") parser.add_argument("--adam_beta1", type=float, default=0.9) parser.add_argument("--adam_beta2", type=float, default=0.999) parser.add_argument("--seed", type=int, default=1216) parser.add_argument("--solver", choices=["adam", "newton", "adam_newton", "fake_zero"], default="adam") parser.add_argument("--save_ckpts", default="", help = "path to save checkpoint file") parser.add_argument("--load_ckpt", default="", help = "path to numpy checkpoint file") parser.add_argument("--start_iter", default=0, type=int, help = "iteration number for start, useful when loading a checkpoint") parser.add_argument("--init_size", default=32, type=int, help = "starting with this size when --use_resize") parser.add_argument("--uniform", action='store_true', help = "disable importance sampling") args = vars(parser.parse_args()) # add some additional parameters # learning rate args['lr'] = 1e-2 args['inception'] = False args['use_tanh'] = True # args['use_resize'] = False if args['maxiter'] == 0: if args['attack'] == "white": args['maxiter'] = 1000 else: if args['dataset'] == "imagenet": if args['untargeted']: args['maxiter'] = 1500 else: args['maxiter'] = 50000 elif args['dataset'] == "mnist": args['maxiter'] = 3000 else: args['maxiter'] = 1000 if args['init_const'] == 0.0: if args['binary_steps'] != 0: args['init_const'] = 0.01 else: args['init_const'] = 0.5 if args['binary_steps'] == 0: args['binary_steps'] = 1 # set up some parameters based on datasets if args['dataset'] == "imagenet": args['inception'] = True args['lr'] = 2e-3 # args['use_resize'] = True # args['save_ckpts'] = True # for mnist, using tanh causes gradient to vanish if args['dataset'] == "mnist": args['use_tanh'] = False # when init_const is not specified, use a reasonable default if args['init_const'] == 0.0: if args['binary_search']: args['init_const'] = 0.01 else: args['init_const'] = 0.5 # setup random seed random.seed(args['seed']) np.random.seed(args['seed']) print(args) main(args) ================================================ FILE: test_attack.py ================================================ ## test_attack.py -- sample code to test attack procedure ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import tensorflow as tf import numpy as np import random import time from setup_cifar import CIFAR, CIFARModel from setup_mnist import MNIST, MNISTModel from setup_inception import ImageNet, InceptionModel from l2_attack import CarliniL2 from l0_attack import CarliniL0 from li_attack import CarliniLi from PIL import Image def show(img, name = "output.png"): """ Show MNSIT digits in the console. """ np.save('img', img) fig = (img + 0.5)*255 fig = fig.astype(np.uint8).squeeze() pic = Image.fromarray(fig) # pic.resize((512,512), resample=PIL.Image.BICUBIC) pic.save(name) remap = " .*#"+"#"*100 img = (img.flatten()+.5)*3 if len(img) != 784: return print("START") for i in range(28): print("".join([remap[int(round(x))] for x in img[i*28:i*28+28]])) def generate_data(data, samples, targeted=True, start=0, inception=False): """ Generate the input data to the attack algorithm. data: the images to attack samples: number of samples to use targeted: if true, construct targeted attacks, otherwise untargeted attacks start: offset into data to use inception: if targeted and inception, randomly sample 100 targets intead of 1000 """ inputs = [] targets = [] for i in range(samples): if targeted: if inception: seq = random.sample(range(1,1001), 10) else: seq = range(data.test_labels.shape[1]) print ('image label:', np.argmax(data.test_labels[start+i])) for j in seq: # skip the original image label if (j == np.argmax(data.test_labels[start+i])) and (inception == False): continue inputs.append(data.test_data[start+i]) targets.append(np.eye(data.test_labels.shape[1])[j]) else: inputs.append(data.test_data[start+i]) targets.append(data.test_labels[start+i]) inputs = np.array(inputs) targets = np.array(targets) return inputs, targets if __name__ == "__main__": with tf.Session() as sess: use_log = False print('Loading model...') # data, model = MNIST(), MNISTModel("models/mnist", sess, use_log) # data, model = MNIST(), MNISTModel("models/mnist-distilled-100", sess, use_log) # data, model = CIFAR(), CIFARModel("models/cifar", sess, use_log) data, model = ImageNet(), InceptionModel(sess, use_log) print('Done...') batch_size = 1 if isinstance(model, InceptionModel): batch_size = 10 attack = CarliniL2(sess, model, batch_size=batch_size, initial_const = 1.0, max_iterations=1000, confidence=0, use_log=use_log) print('Generate data') inputs, targets = generate_data(data, samples=1, targeted=True, start=6, inception=isinstance(model, InceptionModel)) print('Done...') print(inputs.shape) inputs = inputs[0:batch_size] targets = targets[0:batch_size] timestart = time.time() adv = attack.attack(inputs, targets) timeend = time.time() print("Took",timeend-timestart,"seconds to run",len(inputs),"samples.") for i in range(len(adv)): print("Valid:") show(inputs[i], "original_{}.png".format(i)) print("Classification:", np.argsort(model.model.predict(inputs[i:i+1]))[-1:-6:-1]) print("Target:", np.argmax(targets[i])) print("Adversarial:") show(adv[i], "adversarial_{}.png".format(i)) show(adv[i] - inputs[i], "attack_diff.png") print("Classification:", np.argsort(model.model.predict(adv[i:i+1]))[-1:-6:-1]) print("Total distortion:", np.sum((adv[i]-inputs[i])**2)**.5) # t = np.random.randn(28*28).reshape(1,28,28,1) # print(model.model.predict(t)) ================================================ FILE: test_attack_black.py ================================================ ## test_attack.py -- sample code to test attack procedure ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import tensorflow as tf import numpy as np import time from setup_cifar import CIFAR, CIFARModel from setup_mnist import MNIST, MNISTModel from setup_inception import ImageNet, InceptionModel from l2_attack import CarliniL2 from l0_attack import CarliniL0 from li_attack import CarliniLi from l2_attack_black import BlackBoxL2 from PIL import Image def show(img, name = "output.png"): """ Show MNSIT digits in the console. """ np.save('img', img) fig = (img + 0.5)*255 fig = fig.astype(np.uint8).squeeze() pic = Image.fromarray(fig) # pic.resize((512,512), resample=PIL.Image.BICUBIC) pic.save(name) remap = " .*#"+"#"*100 img = (img.flatten()+.5)*3 if len(img) != 784: return print("START") for i in range(28): print("".join([remap[int(round(x))] for x in img[i*28:i*28+28]])) def generate_data(data, samples, targeted=True, start=0, inception=False): """ Generate the input data to the attack algorithm. data: the images to attack samples: number of samples to use targeted: if true, construct targeted attacks, otherwise untargeted attacks start: offset into data to use inception: if targeted and inception, randomly sample 100 targets intead of 1000 """ inputs = [] targets = [] for i in range(samples): if targeted: if inception: seq = random.sample(range(1,1001), 10) else: seq = range(data.test_labels.shape[1]) # print ('image label:', np.argmax(data.test_labels[start+i])) for j in seq: # skip the original image label if (j == np.argmax(data.test_labels[start+i])) and (inception == False): continue inputs.append(data.test_data[start+i]) targets.append(np.eye(data.test_labels.shape[1])[j]) else: inputs.append(data.test_data[start+i]) targets.append(data.test_labels[start+i]) inputs = np.array(inputs) targets = np.array(targets) return inputs, targets if __name__ == "__main__": with tf.Session() as sess: use_log = True # data, model = MNIST(), MNISTModel("models/mnist", sess, use_log) # data, model = CIFAR(), CIFARModel("models/cifar", sess, use_log) data, model = ImageNet(), InceptionModel(sess, use_log) attack = BlackBoxL2(sess, model, batch_size=128, max_iterations=15000, confidence=0, use_log=use_log) inputs, targets = generate_data(data, samples=1, targeted=True, start=6, inception=False) inputs = inputs[1:2] targets = targets[1:2] timestart = time.time() adv = attack.attack(inputs, targets) timeend = time.time() print("Took",timeend-timestart,"seconds to run",len(inputs),"samples.") print("Valid:") show(inputs[0], "original.png") print("Adversarial:") show(adv, "adversarial.png") show(adv - inputs[0], "diff.png") print("Valid Classification:", np.argsort(model.model.predict(inputs[0].reshape((1,) + adv.shape)))[-1:-6:-1]) print("Adversarial Classification:", np.argsort(model.model.predict(adv.reshape((1,) + adv.shape)))[-1:-6:-1]) print("Total distortion:", np.sum((adv-inputs[0])**2)**.5) ================================================ FILE: train_models.py ================================================ ## train_models.py -- train the neural network models for attacking ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. import numpy as np from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Conv2D, MaxPooling2D from keras.optimizers import SGD import tensorflow as tf from setup_mnist import MNIST from setup_cifar import CIFAR import os def train(data, file_name, params, num_epochs=50, batch_size=128, train_temp=1, init=None): """ Standard neural network training procedure. """ model = Sequential() print(data.train_data.shape) model.add(Conv2D(params[0], (3, 3), input_shape=data.train_data.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(params[1], (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Conv2D(params[2], (3, 3))) model.add(Activation('relu')) model.add(Conv2D(params[3], (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Flatten()) model.add(Dense(params[4])) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(params[5])) model.add(Activation('relu')) model.add(Dense(10)) if init != None: model.load_weights(init) def fn(correct, predicted): return tf.nn.softmax_cross_entropy_with_logits(labels=correct, logits=predicted/train_temp) sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True) model.compile(loss=fn, optimizer=sgd, metrics=['accuracy']) model.fit(data.train_data, data.train_labels, batch_size=batch_size, validation_data=(data.validation_data, data.validation_labels), nb_epoch=num_epochs, shuffle=True) if file_name != None: model.save(file_name) return model def train_distillation(data, file_name, params, num_epochs=50, batch_size=128, train_temp=1): """ Train a network using defensive distillation. Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami IEEE S&P, 2016. """ if not os.path.exists(file_name+"_init"): # Train for one epoch to get a good starting point. train(data, file_name+"_init", params, 1, batch_size) # now train the teacher at the given temperature teacher = train(data, file_name+"_teacher", params, num_epochs, batch_size, train_temp, init=file_name+"_init") # evaluate the labels at temperature t predicted = teacher.predict(data.train_data) with tf.Session() as sess: y = sess.run(tf.nn.softmax(predicted/train_temp)) print(y) data.train_labels = y # train the student model at temperature t student = train(data, file_name, params, num_epochs, batch_size, train_temp, init=file_name+"_init") # and finally we predict at temperature 1 predicted = student.predict(data.train_data) print(predicted) if not os.path.isdir('models'): os.makedirs('models') train(CIFAR(), "models/cifar", [64, 64, 128, 128, 256, 256], num_epochs=50) train(MNIST(), "models/mnist", [32, 32, 64, 64, 200, 200], num_epochs=50) train_distillation(MNIST(), "models/mnist-distilled-100", [32, 32, 64, 64, 200, 200], num_epochs=50, train_temp=100) train_distillation(CIFAR(), "models/cifar-distilled-100", [64, 64, 128, 128, 256, 256], num_epochs=50, train_temp=100) ================================================ FILE: verify.py ================================================ ## verify.py -- check the accuracy of a neural network ## ## Copyright (C) IBM Corp, 2017-2018 ## Copyright (C) 2016, Nicholas Carlini . ## ## This program is licenced under the BSD 2-Clause licence, ## contained in the LICENCE file in this directory. from setup_cifar import CIFAR, CIFARModel from setup_mnist import MNIST, MNISTModel from setup_inception import ImageNet, InceptionModel import tensorflow as tf import numpy as np BATCH_SIZE = 1 with tf.Session() as sess: data, model = MNIST(), MNISTModel("models/mnist", sess) data, model = CIFAR(), CIFARModel("models/cifar", sess) data, model = ImageNet(), InceptionModel(sess) x = tf.placeholder(tf.float32, (None, model.image_size, model.image_size, model.num_channels)) y = model.predict(x) r = [] for i in range(0,len(data.test_data),BATCH_SIZE): pred = sess.run(y, {x: data.test_data[i:i+BATCH_SIZE]}) #print(pred) #print('real',data.test_labels[i],'pred',np.argmax(pred)) r.append(np.argmax(pred,1) == np.argmax(data.test_labels[i:i+BATCH_SIZE],1)) print(np.mean(r))