[
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*,cover\n.hypothesis/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# IPython Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# dotenv\n.env\n\n# virtualenv\nvenv/\nENV/\n\n# Spyder project settings\n.spyderproject\n\n# Rope project settings\n.ropeproject\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2017 \n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# Keras-GAN-Animeface-Character\n\nGAN example for Keras. Cuz MNIST is too small and there\nshould an example on something more realistic.\n\n## Some results\n\n### Training for 22 epochs\n\nYoutube Video, click on the image\n\n[![Training for 22 epochs](http://img.youtube.com/vi/YuGFmgCQV8I/0.jpg)](https://www.youtube.com/watch?v=YuGFmgCQV8I)\n\n### Loss graph for 5000 mini-batches\n\n![Loss graph](html_data/loss.png)\n\n1 mini-batch = 64 images.\nDataset = 14490, hence 5000 mini-batches is approximately 22 epochs.\n\n### Some outputs of 5000th min-batch\n\n![Some ouptputs of 5000th mini-batch](html_data/frame_00000499.png)\n\n### Some training images\n\n![Some inputs](html_data/reals.png)\n\n\n## Useful resources, before you go on\n\n* There are great examples on MNIST already. Be sure to check them out.\n    * https://oshearesearch.com/index.php/2016/07/01/mnist-generative-adversarial-model-in-keras/\n    * https://github.com/osh/KerasGAN\n    * https://medium.com/towards-data-science/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0\n* \"How to Train a GAN? Tips and tricks to make GANs work\" is a must read! (GAN Hacks)\n    * The advices were extremely helpful in making this example.\n    * https://github.com/soumith/ganhacks\n* Projects doing the same thing:\n    * https://github.com/jayleicn/animeGAN\n    * https://github.com/tdrussell/IllustrationGAN\n* I used slow implementation for the sake of simplicity. However, the correct way is:\n    * https://ctmakro.github.io/site/on_learning/fast_gan_in_keras.html\n* https://github.com/shekkizh/neuralnetworks.thought-experiments/blob/master/Generative%20Models/GAN/Readme.md\n\n\n## How to run this example\n\n### Setup\n\n* My environment: Python 3.6 + Keras 2.0.4 + Tensorflow 1.x\n    * If you are on Keras 2.0.0, you need to update it otherwise BatchNormalization() will\n      cause bug, saying \"you need to pass float to input\" or something\n      like that from Tensorflow back end.\n* Use [virtualenv](https://docs.python.org/3/tutorial/venv.html) to initialize a similar environment (python and dependencies):\n\n```bash\npip install virtualenv\nvirtualenv -p <PATH_TO_BIN_DIR>/python3.6 venv\nsource venv/bin/activate\npip install -r requirements.txt\n```\n\n* I HATE making a program that has so many command line parameters to pass.\n  Many of the parameters are there in the scripts. Adjust the script as you need.\n  The \"main()\" function is at the bottom of the script as people do in C/C++\n* Most global parameters are defined in args.py.\n    * They are defined as class variables not instance variables so you may have trouble\n      running/training multiple instances of the GAN with different parameters.\n      (which is very unlikely to happen)\n* Download dataset from http://www.nurs.or.jp/~nagadomi/animeface-character-dataset/\n    * Extract it to this directory so that the scipt can find\n      ./animeface-character-dataset/thumb/\n    * Any dataset should work in principle but GAN is sensitive to hyperparameters and may not work\n      on yours. I tuned the parameters for animeface-character-dataset.\n\n### Preprocessing\n* Run the preprocessing script. It saves training time to resize/scale the input than\n  doing those tasks on the fly in the training loop.\n    * ./data.py\n    * The image, when loaded from PNG files, the RGB values have [0, 255]. (uint8 type). data.py will collect the images, resize the images to 64x64 and scale the RGB values so that they will be in [-1.0, 1.0] range.\n    * Data.py will only sample a subset of the dataset if configured to do so. The size of the subset is determined\n      by dataset_sz defined in args.py\n    * The images will be written to data.hdf5.\n        * Made it small to verify the training is working.\n        * You can increase it but you need to adjust the network sizes accordingly.\n    * Again, which files to read is defined in the script at the bottom, not by sys.argv.\n* You need a large enough dataset. Otherwise the discriminator will sort of \"memorize\"\n  the true data and reject all that's generated.\n\n### Training\n* Open gan.py then at the bottom, uncomment train\\_autoenc() if you wish.\n    * This is useful for seeing the generator network's capability to reproduce the input.\n    * The auto-encoder will be trained on input images.\n    * The output will be blurry, as the auto-encoder having mean-squared-error loss.\n      (This is why GAN got invented in the first place!)\n* To run training, modify main() so that train\\_gan() is uncommented.\n* The script will dump reals.png and fakes.png every 10 epoch so that you can see\n  how the training is going.\n* The training takes a while. For this example on Anime Face dataset, it took about 10000 mini-batches\n  to get good results.\n    * If you see only uniform color or \"modern art\" until 2000 then the training is not working!\n* The script also dumps weights every 10 batches. Utilize them to save training time.\n  Weights before diverging is preferred :)\n  Uncomment load\\_weights() in train\\_gan().\n\n### Training tips\nWhat I experienced during my training of GAN.\n* As described in GAN Hacks, discriminator should be ahead of the generator so that\n  the generator can be \"guided\" by the discriminator.\n* If you look at loss graph at https://github.com/osh/KerasGAN,\n  they had gen loss in range of 2 to 4. Their training worked well.\n  The discriminator loss is low, arond 0.1.\n* You'll need trial and error to get the hyper-pameters right\n  so that the training stays in the stable, balanced zone.\n  That includes learning rate of D and G, momentums, etc.\n* The convergence is quite sensitive with LR, beware!\n* If things go well, the discriminator loss for detecting real/fake = dloss0/dloss1 should\n  be less than or around 0.1, which means it is good at telling whether the input is real or fake.\n* If learning rate is too high, the discriminator will diverge and\n  one of the loss will get high and will not fall. Training fails in this case.\n* If you make LR too small, it will only slow the learning and will not prevent other issues\n  such as oscillation. It only needs to be lower than certain threshold that is data dependent.\n* If adjusting LR doesn't work, it could be lack of complexity in the discriminator layer.\n  Add more layers, or some other parameters. It could be anything :( Good luck!\n* On the other hand, generator loss will be relatively higher than discriminator loss.\n  In this script, it oscillates in range 0.1 to 4.\n* If you see any of the D loss staying > 15 (when batch size is 32) the training is screwed.\n* In case of G loss > 15, see if it escapes within 30 batches.\n  If it stays there for too long, it isn't good, I think.\n* In case you're seeing high G loss, it could mean it can't keep up with discriminator.\n  You might need to increase LR. (Must be slower than discriminator though)\n* One final piece of the training I was missing was the parameter in BatchNormalization.\n  I found about it in this link:\n  https://github.com/shekkizh/neuralnetworks.thought-experiments/blob/master/Generative%20Models/GAN/Readme.md\n    * Sort of interesting, in PyTorch, momentum parameter for BatchNorm is 0.1, according to the API documents, while in Keras it is 0.99. I'm not sure if 0.1 in PyTorch actually means 1 - 0.1. I didn't look into PyTorch backend implementation.\n"
  },
  {
    "path": "args.py",
    "content": "#!/usr/bin/env python3\n\nclass Args :\n    # dataset size... Use positive number to sample subset of the full dataset.\n    dataset_sz = -1\n\n    # Archive outputs of training here for animating later.\n    anim_dir = \"anim\"\n\n    # images size we will work on. (sz, sz, 3)\n    sz = 64\n    \n    # alpha, used by leaky relu of D and G networks.\n    alpha_D = 0.2\n    alpha_G = 0.2\n\n    # batch size, during training.\n    batch_sz = 64\n\n    # Length of the noise vector to generate the faces from.\n    # Latent space z\n    noise_shape = (1, 1, 100)\n\n    # GAN training can be ruined any moment if not careful.\n    # Archive some snapshots in this directory.\n    snapshot_dir = \"./snapshots\"\n\n    # dropout probability\n    dropout = 0.3\n\n    # noisy label magnitude\n    label_noise = 0.1\n\n    # history to keep. Slower training but higher quality.\n    history_sz = 8\n\n    genw = \"gen.hdf5\"\n    discw = \"disc.hdf5\"\n\n    # Weight initialization function.\n    #kernel_initializer = 'Orthogonal'\n    #kernel_initializer = 'RandomNormal'\n    # Same as default in Keras, but good for GAN, says\n    # https://github.com/gheinrich/DIGITS-GAN/blob/master/examples/weight-init/README.md#experiments-with-lenet-on-mnist\n    kernel_initializer = 'glorot_uniform'\n\n    # Since DCGAN paper, everybody uses 0.5 and for me, it works the best too.\n    # I tried 0.9, 0.1.\n    adam_beta = 0.5\n\n    # BatchNormalization matters too.\n    bn_momentum = 0.3\n"
  },
  {
    "path": "data.py",
    "content": "#!/usr/bin/env python3\nimport glob\nimport h5py\nimport numpy as np\nimport scipy.misc\nimport random\nimport cv2\n\nfrom args import Args\n\n\n\ndef normalize4gan(im):\n    '''\n    Convert colorspace and\n    cale the input in [-1, 1] range, as described in ganhacks\n    '''\n    #im = cv2.cvtColor(im, cv2.COLOR_RGB2YCR_CB).astype(np.float32)\n    # HSV... not helpful.\n    im = im.astype(np.float32)\n    im /= 128.0\n    im -= 1.0 # now in [-1, 1]\n    return im\n\n\n\ndef denormalize4gan(im):\n    '''\n    Does opposite of normalize4gan:\n    [-1, 1] to [0, 255].\n    Warning: input im is modified in-place!\n    '''\n    im += 1.0 # in [0, 2]\n    im *= 127.0 # in [0, 255]\n    return im.astype(np.uint8)\n\n\n\ndef make_hdf5(ofname, wildcard):\n    '''\n    Preprocess files given by wildcard and save them in hdf5 file, as ofname.\n    '''\n    pool = list(glob.glob(wildcard))\n    if Args.dataset_sz <= 0:\n        fnames = pool\n    else:\n        fnames = []\n        for i in range(Args.dataset_sz):\n            # possible duplicate but don't care\n            fnames.append(random.choice(pool))\n\n    with h5py.File(ofname, \"w\") as f:\n        faces = f.create_dataset(\"faces\", (len(fnames), Args.sz, Args.sz, 3), dtype='f')\n\n        for i, fname in enumerate(fnames):\n            print(fname)\n            im = scipy.misc.imread(fname, mode='RGB') # some have alpha channel\n            im = scipy.misc.imresize(im, (Args.sz, Args.sz))\n            faces[i] = normalize4gan(im)\n\n\n\ndef test(hdff):\n    '''\n    Reads in hdf file and check if pixels are scaled in [-1, 1] range.\n    '''\n    with h5py.File(hdff, \"r\") as f:\n        X = f.get(\"faces\")\n        print(np.min(X[:,:,:,0]))\n        print(np.max(X[:,:,:,0]))\n        print(np.min(X[:,:,:,1]))\n        print(np.max(X[:,:,:,1]))\n        print(np.min(X[:,:,:,2]))\n        print(np.max(X[:,:,:,2]))\n        print(\"Dataset size:\", len(X))\n        assert np.max(X) <= 1.0\n        assert np.min(X) >= -1.0\n\n\n\nif __name__ == \"__main__\" :\n    # Thankfully the dataset is in PNG, not JPEG.\n    # Anime style suffers from significant quality degradation in JPEG.\n    make_hdf5(\"data.hdf5\", \"animeface-character-dataset/thumb/*/*.png\")\n    #make_hdf5(\"data.hdf5\", \"animeface-character-dataset/thumb/025*/*.png\")\n\n    # Uncomment and run test, if you want.\n    test(\"data.hdf5\")\n"
  },
  {
    "path": "discrimination.py",
    "content": "from keras import backend as K\nfrom keras.engine import InputSpec, Layer\nfrom keras import initializers, regularizers, constraints\n\n# From a PR that is not pulled into Keras\n# https://github.com/fchollet/keras/pull/3677\n# I updated the code to work on Keras 2.x\n\nclass MinibatchDiscrimination(Layer):\n    \"\"\"Concatenates to each sample information about how different the input\n    features for that sample are from features of other samples in the same\n    minibatch, as described in Salimans et. al. (2016). Useful for preventing\n    GANs from collapsing to a single output. When using this layer, generated\n    samples and reference samples should be in separate batches.\n\n    # Example\n\n    ```python\n        # apply a convolution 1d of length 3 to a sequence with 10 timesteps,\n        # with 64 output filters\n        model = Sequential()\n        model.add(Convolution1D(64, 3, border_mode='same', input_shape=(10, 32)))\n        # now model.output_shape == (None, 10, 64)\n\n        # flatten the output so it can be fed into a minibatch discrimination layer\n        model.add(Flatten())\n        # now model.output_shape == (None, 640)\n\n        # add the minibatch discrimination layer\n        model.add(MinibatchDiscrimination(5, 3))\n        # now model.output_shape = (None, 645)\n    ```\n\n    # Arguments\n        nb_kernels: Number of discrimination kernels to use\n            (dimensionality concatenated to output).\n        kernel_dim: The dimensionality of the space where closeness of samples\n            is calculated.\n        init: name of initialization function for the weights of the layer\n            (see [initializations](../initializations.md)),\n            or alternatively, Theano function to use for weights initialization.\n            This parameter is only relevant if you don't pass a `weights` argument.\n        weights: list of numpy arrays to set as initial weights.\n        W_regularizer: instance of [WeightRegularizer](../regularizers.md)\n            (eg. L1 or L2 regularization), applied to the main weights matrix.\n        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),\n            applied to the network output.\n        W_constraint: instance of the [constraints](../constraints.md) module\n            (eg. maxnorm, nonneg), applied to the main weights matrix.\n        input_dim: Number of channels/dimensions in the input.\n            Either this argument or the keyword argument `input_shape`must be\n            provided when using this layer as the first layer in a model.\n\n    # Input shape\n        2D tensor with shape: `(samples, input_dim)`.\n\n    # Output shape\n        2D tensor with shape: `(samples, input_dim + nb_kernels)`.\n\n    # References\n        - [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498)\n    \"\"\"\n\n    def __init__(self, nb_kernels, kernel_dim, init='glorot_uniform', weights=None,\n                 W_regularizer=None, activity_regularizer=None,\n                 W_constraint=None, input_dim=None, **kwargs):\n        self.init = initializers.get(init)\n        self.nb_kernels = nb_kernels\n        self.kernel_dim = kernel_dim\n        self.input_dim = input_dim\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.activity_regularizer = regularizers.get(activity_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n\n        self.initial_weights = weights\n        self.input_spec = [InputSpec(ndim=2)]\n\n        if self.input_dim:\n            kwargs['input_shape'] = (self.input_dim,)\n        super(MinibatchDiscrimination, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 2\n\n        input_dim = input_shape[1]\n        self.input_spec = [InputSpec(dtype=K.floatx(),\n                                     shape=(None, input_dim))]\n\n        self.W = self.add_weight(shape=(self.nb_kernels, input_dim, self.kernel_dim),\n            initializer=self.init,\n            name='kernel',\n            regularizer=self.W_regularizer,\n            trainable=True,\n            constraint=self.W_constraint)\n\n        # Set built to true.\n        super(MinibatchDiscrimination, self).build(input_shape)\n\n    def call(self, x, mask=None):\n        activation = K.reshape(K.dot(x, self.W), (-1, self.nb_kernels, self.kernel_dim))\n        diffs = K.expand_dims(activation, 3) - K.expand_dims(K.permute_dimensions(activation, [1, 2, 0]), 0)\n        abs_diffs = K.sum(K.abs(diffs), axis=2)\n        minibatch_features = K.sum(K.exp(-abs_diffs), axis=2)\n        return K.concatenate([x, minibatch_features], 1)\n\n    def compute_output_shape(self, input_shape):\n        assert input_shape and len(input_shape) == 2\n        return input_shape[0], input_shape[1]+self.nb_kernels\n\n    def get_config(self):\n        config = {'nb_kernels': self.nb_kernels,\n                  'kernel_dim': self.kernel_dim,\n                  'init': self.init.__name__,\n                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,\n                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,\n                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,\n                  'input_dim': self.input_dim}\n        base_config = super(MinibatchDiscrimination, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n"
  },
  {
    "path": "gan.py",
    "content": "#!/usr/bin/env python3\nimport os\nimport sys\nimport numpy as np\nimport random\nfrom keras import models\nfrom keras import optimizers\nfrom keras.layers import Input\nfrom keras.optimizers import Adam, Adagrad, Adadelta, Adamax, SGD\nfrom keras.callbacks import CSVLogger\nimport scipy\nimport h5py\nfrom args import Args\nfrom data import denormalize4gan\n#from layers import bilinear2x\nfrom discrimination import MinibatchDiscrimination\nfrom nets import build_discriminator, build_gen, build_enc\n\n#import tensorflow as tf\n#import keras\n#keras.backend.get_session().run(tf.initialize_all_variables())\n\n\n\ndef sample_faces( faces ):\n    reals = []\n    for i in range( Args.batch_sz ) :\n        j = random.randrange( len(faces) )\n        face = faces[ j ]\n        reals.append( face )\n    reals = np.array(reals)\n    return reals\n\n\n\ndef binary_noise(cnt):\n    # Distribution of noise matters.\n    # If you use single ranf that spans [0, 1], training will not work.\n    # Well, for me at least.\n    # Either normal or ranf works for me but be sure to use them with randrange(2) or something.\n    #noise = np.random.normal( scale=Args.label_noise, size=((Args.batch_sz,) + Args.noise_shape) )\n\n    # Note about noise rangel.\n    # 0, 1 noise vs -1, 1 noise. -1, 1 seems to be better and stable.\n\n    noise = Args.label_noise * np.random.ranf((cnt,) + Args.noise_shape) # [0, 0.1]\n    noise -= 0.05 # [-0.05, 0.05]\n    noise += np.random.randint(0, 2, size=((cnt,) + Args.noise_shape))\n\n    noise -= 0.5\n    noise *= 2\n    return noise\n\n\n\ndef sample_fake( gen ) :\n    noise = binary_noise(Args.batch_sz)\n    fakes = gen.predict(noise)\n    return fakes, noise\n\n\n\ndef dump_batch(imgs, cnt, ofname):\n    '''\n    Merges cnt x cnt generated images into one big image.\n    Use the command\n    $ feh dump.png --reload 1\n    to refresh image peroidically during training!\n    '''\n    assert Args.batch_sz >= cnt * cnt\n\n    rows = []\n\n    for i in range( cnt ) :\n        cols = []\n        for j in range(cnt*i, cnt*i+cnt):\n            cols.append( imgs[j] )\n        rows.append( np.concatenate(cols, axis=1) )\n\n    alles = np.concatenate( rows, axis=0 )\n    alles = denormalize4gan( alles )\n    #alles = scipy.misc.imresize(alles, 200) # uncomment to scale\n    scipy.misc.imsave( ofname, alles )\n\n\n\ndef build_networks():\n    shape = (Args.sz, Args.sz, 3)\n\n    # Learning rate is important.\n    # Optimizers are important too, try experimenting them yourself to fit your dataset.\n    # I recommend you read DCGAN paper.\n\n    # Unlike gan hacks, sgd doesn't seem to work well.\n    # DCGAN paper states that they used Adam for both G and D.\n    #opt  = optimizers.SGD(lr=0.0001, decay=0.0, momentum=0.9, nesterov=True)\n    #dopt = optimizers.SGD(lr=0.0001, decay=0.0, momentum=0.9, nesterov=True)\n\n    # lr=0.010. Looks good, statistically (low d loss, higher g loss)\n    # but too much for the G to create face.\n    # If you see only one color 'flood fill' during training for about 10 batches or so,\n    # training is failing. If you see only a few colors (instead of colorful noise)\n    # then lr is too high for the opt and G will not have chance to form face.\n    #dopt = Adam(lr=0.010, beta_1=0.5)\n    #opt  = Adam(lr=0.001, beta_1=0.5)\n\n    # vague faces @ 500\n    # Still can't get higher frequency component.\n    #dopt = Adam(lr=0.0010, beta_1=0.5)\n    #opt  = Adam(lr=0.0001, beta_1=0.5)\n\n    # better faces @ 500\n    # but mode collapse after that, probably due to learning rate being too high.\n    # opt.lr = dopt.lr / 10 works nicely. I found this with trial and error.\n    # now same lr, as we are using history to train D multiple times.\n    # I don't exactly understand how decay parameter in Adam works. Certainly not exponential.\n    # Actually faster than exponential, when I look at the code and plot it in Excel.\n    dopt = Adam(lr=0.0002, beta_1=Args.adam_beta)\n    opt  = Adam(lr=0.0001, beta_1=Args.adam_beta)\n\n    # too slow\n    # Another thing about LR.\n    # If you make it small, it will only optimize slowly.\n    # LR only has to be smaller than certain threshold that is data dependent.\n    # (related to the largest gradient that prevents optimization)\n    #dopt = Adam(lr=0.000010, beta_1=0.5)\n    #opt  = Adam(lr=0.000001, beta_1=0.5)\n\n    # generator part\n    gen = build_gen( shape )\n    # loss function doesn't seem to matter for this one, as it is not directly trained\n    gen.compile(optimizer=opt, loss='binary_crossentropy')\n    gen.summary()\n\n    # discriminator part\n    disc = build_discriminator( shape )\n    disc.compile(optimizer=dopt, loss='binary_crossentropy')\n    disc.summary()\n\n    # GAN stack\n    # https://ctmakro.github.io/site/on_learning/fast_gan_in_keras.html is the faster way.\n    # Here, for simplicity, I use slower way (slower due to duplicate computation).\n    noise = Input( shape=Args.noise_shape )\n    gened = gen( noise )\n    result = disc( gened )\n    gan = models.Model( inputs=noise, outputs=result )\n    gan.compile(optimizer=opt, loss='binary_crossentropy')\n    gan.summary()\n\n    return gen, disc, gan\n\n\n\ndef train_autoenc( dataf ):\n    '''\n    Train an autoencoder first to see if your network is large enough.\n    '''\n    f = h5py.File( dataf, 'r' )\n    faces = f.get( 'faces' )\n\n    opt = Adam(lr=0.001)\n\n    shape = (Args.sz, Args.sz, 3)\n    enc = build_enc( shape )\n    enc.compile(optimizer=opt, loss='mse')\n    enc.summary()\n\n    # generator part\n    gen = build_gen( shape )\n    # generator is not directly trained. Optimizer and loss doesn't matter too much.\n    gen.compile(optimizer=opt, loss='mse')\n    gen.summary()\n\n    face = Input( shape=shape )\n    vector = enc(face)\n    recons = gen(vector)\n    autoenc = models.Model( inputs=face, outputs=recons )\n    autoenc.compile(optimizer=opt, loss='mse')\n\n    epoch = 0\n    while epoch < 200 :\n        for i in range(10) :\n            reals = sample_faces( faces  )\n            fakes, noises = sample_fake( gen )\n            loss = autoenc.train_on_batch( reals, reals )\n            epoch += 1\n            print(epoch, loss)\n        fakes = autoenc.predict(reals)\n        dump_batch(fakes, 4, \"fakes.png\")\n        dump_batch(reals, 4, \"reals.png\")\n    gen.save_weights(Args.genw)\n    enc.save_weights(Args.discw)\n    print(\"Saved\", Args.genw, Args.discw)\n\n\n\ndef load_weights(model, wf):\n    '''\n    I find error message in load_weights hard to understand sometimes.\n    '''\n    try:\n        model.load_weights(wf)\n    except:\n        print(\"failed to load weight, network changed or corrupt hdf5\", wf, file=sys.stderr)\n        sys.exit(1)\n\n\n\ndef train_gan( dataf ) :\n    gen, disc, gan = build_networks()\n\n    # Uncomment these, if you want to continue training from some snapshot.\n    # (or load pretrained generator weights)\n    #load_weights(gen, Args.genw)\n    #load_weights(disc, Args.discw)\n\n    logger = CSVLogger('loss.csv') # yeah, you can use callbacks independently\n    logger.on_train_begin() # initialize csv file\n    with h5py.File( dataf, 'r' ) as f :\n        faces = f.get( 'faces' )\n        run_batches(gen, disc, gan, faces, logger, range(5000))\n    logger.on_train_end()\n\n\n\ndef run_batches(gen, disc, gan, faces, logger, itr_generator):\n    history = [] # need this to prevent G from shifting from mode to mode to trick D.\n    train_disc = True\n    for batch in itr_generator:\n        # Using soft labels here.\n        lbl_fake = Args.label_noise * np.random.ranf(Args.batch_sz)\n        lbl_real = 1 - Args.label_noise * np.random.ranf(Args.batch_sz)\n\n        fakes, noises = sample_fake( gen )\n        reals = sample_faces( faces )\n        # Add noise...\n        # My dataset works without this.\n        #reals += 0.5 * np.exp(-batch/100) * np.random.normal( size=reals.shape )\n\n        if batch % 10 == 0 :\n            if len(history) > Args.history_sz:\n                history.pop(0) # evict oldest\n            history.append( (reals, fakes) )\n\n        gen.trainable = False\n        #for reals, fakes in history:\n        d_loss1 = disc.train_on_batch( reals, lbl_real )\n        d_loss0 = disc.train_on_batch( fakes, lbl_fake )\n        gen.trainable = True\n       \n        #if d_loss1 > 15.0 or d_loss0 > 15.0 :\n        # artificial training of one of G or D based on\n        # statistics is not good at all.\n\n        # pretrain train discriminator only\n        if batch < 20 :\n            print( batch, \"d0:{} d1:{}\".format( d_loss0, d_loss1 ) )\n            continue\n\n        disc.trainable = False\n        g_loss = gan.train_on_batch( noises, lbl_real ) # try to trick the classifier.\n        disc.trainable = True\n\n        # To escape this loop, both D and G should be trained so that\n        # D begins to mark everything that's wrong that G has done.\n        # Otherwise G will only change locally and fail to escape the minima.\n        #train_disc = True if g_loss < 15 else False\n\n        print( batch, \"d0:{} d1:{}   g:{}\".format( d_loss0, d_loss1, g_loss ) )\n\n        # save weights every 10 batches\n        if batch % 10 == 0 and batch != 0 :\n            end_of_batch_task(batch, gen, disc, reals, fakes)\n            row = {\"d_loss0\": d_loss0, \"d_loss1\": d_loss1, \"g_loss\": g_loss}\n            logger.on_epoch_end(batch, row)\n\n\n\n_bits = binary_noise(Args.batch_sz)\ndef end_of_batch_task(batch, gen, disc, reals, fakes):\n    try :\n        # Dump how the generator is doing.\n        # Animation dump\n        dump_batch(reals, 4, \"reals.png\")\n        dump_batch(fakes, 4, \"fakes.png\") # to check how noisy the image is\n        frame = gen.predict(_bits)\n        animf = os.path.join(Args.anim_dir, \"frame_{:08d}.png\".format(int(batch/10)))\n        dump_batch(frame, 4, animf)\n        dump_batch(frame, 4, \"frame.png\")\n\n        serial = int(batch / 10) % 10\n        prefix = os.path.join(Args.snapshot_dir, str(serial) + \".\")\n\n        print(\"Saving weights\", serial)\n        gen.save_weights(prefix + Args.genw)\n        disc.save_weights(prefix + Args.discw)\n    except KeyboardInterrupt :\n        print(\"Saving, don't interrupt with Ctrl+C!\", serial)\n        # recursion to surely save everything haha\n        end_of_batch_task(batch, gen, disc, reals, fakes)\n        raise\n\n\n\ndef generate( genw, cnt ):\n    shape = (Args.sz, Args.sz, 3)\n    gen = build_gen( shape )\n    gen.compile(optimizer='sgd', loss='mse')\n    load_weights(gen, Args.genw)\n\n    generated = gen.predict(binary_noise(Args.batch_sz))\n    # Unoffset, in batch.\n    # Must convert back to unit8 to stop color distortion.\n    generated = denormalize4gan(generated)\n\n    for i in range(cnt):\n        ofname = \"{:04d}.png\".format(i)\n        scipy.misc.imsave( ofname, generated[i] )\n\n\n\ndef main( argv ) :\n    if not os.path.exists(Args.snapshot_dir) :\n        os.mkdir(Args.snapshot_dir)\n    if not os.path.exists(Args.anim_dir) :\n        os.mkdir(Args.anim_dir)\n\n    # test the capability of generator network through autoencoder test.\n    # The argument is that if the generator network can memorize the inputs then\n    # it should be enough to GAN-generate stuff.\n    # Pretraining gen isn't that useful in gan training as\n    # the untrained discriminator will soon ruin everything.\n    #train_autoenc( \"data.hdf5\" )\n\n    # train GAN with inputs in data.hdf5\n    train_gan( \"data.hdf5\" )\n\n    # Lets generate stuff\n    #generate( \"gen.hdf5\", 256 )\n\n\n\nif __name__ == '__main__':\n    main(sys.argv)\n"
  },
  {
    "path": "layers.py",
    "content": "#!/usr/bin/env python3\nfrom keras.layers.convolutional import Conv2DTranspose\nfrom keras.initializers import Constant\nimport numpy as np\n\n\n\ndef upsample_filt(size):\n    factor = (size + 1) // 2\n    if size % 2 == 1:\n        center = factor - 1\n    else:\n        center = factor - 0.5\n    og = np.ogrid[:size, :size]\n    return (1 - abs(og[0] - center) / factor) * (1 - abs(og[1] - center) / factor)\n\n\ndef bilinear_upsample_weights(factor, number_of_classes):\n    filter_size = factor*2 - factor%2\n    weights = np.zeros((filter_size, filter_size, number_of_classes, number_of_classes),\n                       dtype=np.float32)\n    upsample_kernel = upsample_filt(filter_size)\n    for i in range(number_of_classes):\n        weights[:, :, i, i] = upsample_kernel\n    return weights\n\n\ndef bilinear2x(x, nfilters):\n\t'''\n    Ugh, I don't like making layers.\n    My credit goes to: https://kivantium.net/keras-bilinear\n    '''\n\treturn Conv2DTranspose(nfilters, (4, 4),\n        strides=(2, 2),\n        padding='same',\n\t\tkernel_initializer=Constant(bilinear_upsample_weights(2, nfilters)))(x)\n"
  },
  {
    "path": "make_mp4.sh",
    "content": "#!/bin/bash\n\n# -r specifies FPS, of the video.\n# -s specifies video size.\n# framerate is the rate the next image will show... sort of confusing with -r but\n# if you want to show one image long, use this option.\n# I might have misunderstood, refer to man page for the options.\n#-framerate 2 \\\n\nffmpeg \\\n    -f image2 \\\n    -i anim/frame_%08d.png \\\n    -r 30 \\\n    -crf 5 -pix_fmt yuv420p \\\n    -vcodec libx264 \\\n    anim.mp4\n"
  },
  {
    "path": "nets.py",
    "content": "#!/usr/bin/env python3\nimport os\nimport sys\nimport numpy as np\nimport random\nfrom keras import models\nfrom keras import optimizers\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.layers.convolutional import Conv2D, Conv2DTranspose, UpSampling2D\nfrom keras.layers.pooling import MaxPooling2D\nfrom keras.layers.core import Dense, Activation, Flatten, Reshape, Dropout\nfrom keras.layers import Input\nfrom keras.optimizers import Adam, Adagrad, Adadelta, Adamax, SGD\nfrom keras.callbacks import CSVLogger\n# GAN doesn't like spare gradients (says ganhack). LeakyReLU better.\nfrom keras.layers.advanced_activations import LeakyReLU\nimport scipy\nimport h5py\nfrom args import Args\nfrom data import denormalize4gan\nfrom layers import bilinear2x\nfrom discrimination import MinibatchDiscrimination\n\n#import tensorflow as tf\n#import keras\n#keras.backend.get_session().run(tf.initialize_all_variables())\n\n\n\ndef build_enc( shape ) :\n    return build_discriminator(shape, build_disc=False)\n\n\n\ndef build_discriminator( shape, build_disc=True ) :\n    '''\n    Build discriminator.\n    Set build_disc=False to build an encoder network to test\n    the encoding/discrimination capability with autoencoder...\n    '''\n    def conv2d( x, filters, shape=(4, 4), **kwargs ) :\n        '''\n        I don't want to write lengthy parameters so I made a short hand function.\n        '''\n        x = Conv2D( filters, shape, strides=(2, 2),\n            padding='same',\n            kernel_initializer=Args.kernel_initializer,\n            **kwargs )( x )\n        #x = MaxPooling2D()( x )\n        x = BatchNormalization(momentum=Args.bn_momentum)( x )\n        x = LeakyReLU(alpha=Args.alpha_D)( x )\n        return x\n\n    # https://github.com/tdrussell/IllustrationGAN\n    # As proposed by them, unlike GAN hacks, MaxPooling works better for anime dataset it seems.\n    # However, animeGAN doesn't use it so I'll keep it more similar to DCGAN.\n\n    face = Input( shape=shape )\n    x = face\n\n    # Warning: Don't batchnorm the first set of Conv2D.\n    x = Conv2D( 64, (4, 4), strides=(2, 2),\n        padding='same',\n        kernel_initializer=Args.kernel_initializer )( x )\n    x = LeakyReLU(alpha=Args.alpha_D)( x )\n    # 32x32\n\n    x = conv2d( x, 128 )\n    # 16x16\n\n    x = conv2d( x, 256 )\n    # 8x8\n\n    x = conv2d( x, 512 )\n    # 4x4\n\n    if build_disc:\n        x = Flatten()(x)\n        # add 16 features. Run 1D conv of size 3.\n        #x = MinibatchDiscrimination(16, 3)( x )\n\n        #x = Dense(1024, kernel_initializer=Args.kernel_initializer)( x )\n        #x = LeakyReLU(alpha=Args.alpha_D)( x )\n\n        # 1 when \"real\", 0 when \"fake\".\n        x = Dense(1, activation='sigmoid',\n            kernel_initializer=Args.kernel_initializer)( x )\n        return models.Model( inputs=face, outputs=x )\n    else:\n        # build encoder.\n        x = Conv2D(Args.noise_shape[2], (4, 4), activation='tanh')(x)\n        return models.Model( inputs=face, outputs=x )\n\n\n\ndef build_gen( shape ) :\n    def deconv2d( x, filters, shape=(4, 4) ) :\n        '''\n        Conv2DTransposed gives me checkerboard artifact...\n        Select one of the 3.\n        '''\n        # Simpe Conv2DTranspose\n        # Not good, compared to upsample + conv2d below.\n        x= Conv2DTranspose( filters, shape, padding='same',\n            strides=(2, 2), kernel_initializer=Args.kernel_initializer )(x)\n\n        # simple and works\n        #x = UpSampling2D( (2, 2) )( x )\n        #x = Conv2D( filters, shape, padding='same' )( x )\n\n        # Bilinear2x... Not sure if it is without bug, not tested yet.\n        # Tend to make output blurry though\n        #x = bilinear2x( x, filters )\n        #x = Conv2D( filters, shape, padding='same' )( x )\n\n        x = BatchNormalization(momentum=Args.bn_momentum)( x )\n        x = LeakyReLU(alpha=Args.alpha_G)( x )\n        return x\n\n    # https://github.com/tdrussell/IllustrationGAN  z predictor...?\n    # might help. Not sure.\n\n    noise = Input( shape=Args.noise_shape )\n    x = noise\n    # 1x1x256\n    # noise is not useful for generating images.\n\n    x= Conv2DTranspose( 512, (4, 4),\n        kernel_initializer=Args.kernel_initializer )(x)\n    x = BatchNormalization(momentum=Args.bn_momentum)( x )\n    x = LeakyReLU(alpha=Args.alpha_G)( x )\n    # 4x4\n    x = deconv2d( x, 256 )\n    # 8x8\n    x = deconv2d( x, 128 )\n    # 16x16\n    x = deconv2d( x, 64 )\n    # 32x32\n\n    # Extra layer\n    x = Conv2D( 64, (3, 3), padding='same',\n        kernel_initializer=Args.kernel_initializer )( x )\n    x = BatchNormalization(momentum=Args.bn_momentum)( x )\n    x = LeakyReLU(alpha=Args.alpha_G)( x )\n    # 32x32\n\n    x= Conv2DTranspose( 3, (4, 4), padding='same', activation='tanh',\n        strides=(2, 2), kernel_initializer=Args.kernel_initializer )(x)\n    # 64x64\n\n    return models.Model( inputs=noise, outputs=x )\n"
  },
  {
    "path": "requirements.txt",
    "content": "absl-py==0.3.0\nastor==0.7.1\nbackports.weakref==1.0rc1\nbleach==1.5.0\nenum34==1.1.6\ngast==0.2.0\ngrpcio==1.13.0\nh5py==2.8.0\nhtml5lib==0.9999999\nKeras==2.0.4\nMarkdown==2.6.11\nnumpy==1.15.0\nopencv-python==3.3.0.9\nPillow==5.2.0\nprotobuf==3.6.0\nPyYAML==3.13\nscipy==1.1.0\nsix==1.11.0\ntensorboard==1.9.0\ntensorflow==1.9.0\ntensorflow-tensorboard==1.5.1\ntermcolor==1.1.0\nTheano==1.0.2\nWerkzeug==0.14.1"
  }
]