Full Code of EdwardTyantov/ultrasound-nerve-segmentation for AI

Repository: EdwardTyantov/ultrasound-nerve-segmentation
Branch: master
Commit: 84a614009cdc
Files: 15
Total size: 70.1 KB

Directory structure:
gitextract_l79q2ook/

├── README.md
├── __init__.py
├── augmentation.py
├── average_ensembles.py
├── current.py
├── current_ensemble.py
├── data.py
├── keras_plus.py
├── metric.py
├── submission.py
├── train.py
├── train_generator.py
├── train_kfold.py
├── u_model.py
└── utils.py

================================================
FILE CONTENTS
================================================

================================================
FILE: README.md
================================================
# Ultrasound nerve segmentation using Keras (1.0.7)
Kaggle Ultrasound Nerve Segmentation competition [Keras]

#Install (Ubuntu {14,16}, GPU)

cuDNN required.

###Theano
- http://deeplearning.net/software/theano/install_ubuntu.html#install-ubuntu
- sudo pip install pydot-ng

In ~/.theanorc
```
[global]
device = gpu0
[dnn]
enabled = True
```

###Keras
- sudo apt-get install libhdf5-dev
- sudo pip install h5py
- sudo pip install pydot
- sudo pip install nose_parameterized
- sudo pip install keras

In ~/.keras/keras.json (it's very important, the project was running on theano backend, and some issues are possible in case of TensorFlow)
```
{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "theano"
}
```

###Python deps
 - sudo apt-get install python-opencv
 - sudo apt-get install python-sklearn

#Prepare

Place train and test data into '../train' and '../test' folders accordingly.

```
mkdir np_data
python data.py
```

#Training

Single model training.
```
python train.py
```
Results will be generatated in "res/" folder. res/unet.hdf5 - best model

Generate submission:
```
python submission.py
```

Generate predection with a model in res/unet.hdf5
``` 
python current.py
```

#Model

Motivation's explained in my internal pres (slides: http://www.slideshare.net/Eduardyantov/ultrasound-segmentation-kaggle-review)

I used U-net like architecture (http://arxiv.org/abs/1505.04597). Main differences:
 - inception blocks instead of VGG like
 - Conv with stride instead of MaxPooling
 - Dropout, p=0.5
 - skip connections from encoder to decoder layers with residual blocks
 - BatchNorm everywhere
 - 2 heads training: auxiliary branch for scoring nerve presence (in the middle of the network), one branch for segmentation
 - ELU activation
 - sigmoid activation in output 
 - Adam optimizer, without weight regularization in layers
 - Dice coeff loss, average per batch, without smoothing
 - output layers - sigmoid activation
 - batch_size=64,128 (for GeForce 1080 and Titan X respectively)

Augmentation:
 - flip x,y
 - random zoom
 - random channel shift
 - elastic transormation didn't help in this configuration

Augmentation generator (generate augmented data on the fly for each epoch) didn't improve the score. 
For prediction augmented images were used.

Validation:

For some reason validation split by patient (which is proper in this competition) didn't work for me, probably due to bug in the code. So I used random split.

Final prediction uses probability of a nerve presence: p_nerve = (p_score + p_segment)/2, where p_segment based on number of output pixels in the mask.

#Results and technical aspects
- On GPU Titan X an epoch took about 6 minutes. Training early stops at 15-30 epochs.
- For batch_size=64 6Gb GPU memory is required.
- Best single model achieved 0.694 LB score.
- An ensemble of 6 different k-fold ensembles (k=5,6,8) scored 0.70399

#Credits
This code was originally based on https://github.com/jocicmarko/ultrasound-nerve-segmentation/


================================================
FILE: __init__.py
================================================


================================================
FILE: augmentation.py
================================================
import sys, os
import numpy as np
from keras.preprocessing.image import (transform_matrix_offset_center, apply_transform, Iterator,
                                       random_channel_shift, flip_axis)
from scipy.ndimage.interpolation import map_coordinates
from scipy.ndimage.filters import gaussian_filter


_dir = os.path.join(os.path.realpath(os.path.dirname(__file__)), '')
data_path = os.path.join(_dir, '../')
aug_data_path = os.path.join(_dir, 'aug_data')
aug_pattern = os.path.join(aug_data_path, 'train_img_%d.npy')
aug_mask_pattern = os.path.join(aug_data_path, 'train_mask_%d.npy')


def random_zoom(x, y, zoom_range, row_index=1, col_index=2, channel_index=0,
                fill_mode='nearest', cval=0.):
    if len(zoom_range) != 2:
        raise Exception('zoom_range should be a tuple or list of two floats. '
                        'Received arg: ', zoom_range)

    if zoom_range[0] == 1 and zoom_range[1] == 1:
        zx, zy = 1, 1
    else:
        zx, zy = np.random.uniform(zoom_range[0], zoom_range[1], 2)
    zoom_matrix = np.array([[zx, 0, 0],
                            [0, zy, 0],
                            [0, 0, 1]])

    h, w = x.shape[row_index], x.shape[col_index]
    transform_matrix = transform_matrix_offset_center(zoom_matrix, h, w)
    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
    y = apply_transform(y, transform_matrix, channel_index, fill_mode, cval)
    return x, y


def random_rotation(x, y, rg, row_index=1, col_index=2, channel_index=0,
                    fill_mode='nearest', cval=0.):
    theta = np.pi / 180 * np.random.uniform(-rg, rg)
    rotation_matrix = np.array([[np.cos(theta), -np.sin(theta), 0],
                                [np.sin(theta), np.cos(theta), 0],
                                [0, 0, 1]])

    h, w = x.shape[row_index], x.shape[col_index]
    transform_matrix = transform_matrix_offset_center(rotation_matrix, h, w)
    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
    y = apply_transform(y, transform_matrix, channel_index, fill_mode, cval)
    return x, y


def random_shear(x, y, intensity, row_index=1, col_index=2, channel_index=0,
                 fill_mode='constant', cval=0.):
    shear = np.random.uniform(-intensity, intensity)
    shear_matrix = np.array([[1, -np.sin(shear), 0],
                             [0, np.cos(shear), 0],
                             [0, 0, 1]])

    h, w = x.shape[row_index], x.shape[col_index]
    transform_matrix = transform_matrix_offset_center(shear_matrix, h, w)
    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
    y = apply_transform(y, transform_matrix, channel_index, fill_mode, cval)
    return x, y


class CustomNumpyArrayIterator(Iterator):

    def __init__(self, X, y, image_data_generator,
                 batch_size=32, shuffle=False, seed=None,
                 dim_ordering='th'):
        self.X = X
        self.y = y
        self.image_data_generator = image_data_generator
        self.dim_ordering = dim_ordering
        super(CustomNumpyArrayIterator, self).__init__(X.shape[0], batch_size, shuffle, seed)


    def next(self):
        with self.lock:
            index_array, _, current_batch_size = next(self.index_generator)
        batch_x = np.zeros(tuple([current_batch_size] + list(self.X.shape)[1:]))
        batch_y_1, batch_y_2 = [], []
        for i, j in enumerate(index_array):
            x = self.X[j]
            y1 = self.y[0][j]
            y2 = self.y[1][j]
            _x, _y1 = self.image_data_generator.random_transform(x.astype('float32'), y1.astype('float32'))
            batch_x[i] = _x
            batch_y_1.append(_y1)
            batch_y_2.append(y2)
        return batch_x, [np.array(batch_y_1), np.array(batch_y_2)]
    

class CustomImageDataGenerator(object):
    def __init__(self, zoom_range=(1,1), channel_shift_range=0, horizontal_flip=False, vertical_flip=False,
                 rotation_range=0,
                 width_shift_range=0.,
                 height_shift_range=0.,
                 shear_range=0.,
                 elastic=None,
):
        self.zoom_range = zoom_range
        self.channel_shift_range = channel_shift_range
        self.horizontal_flip = horizontal_flip
        self.vertical_flip = vertical_flip
        self.rotation_range = rotation_range
        self.width_shift_range = width_shift_range
        self.height_shift_range = height_shift_range
        self.shear_range = shear_range
        self.elastic = elastic
    
    def random_transform(self, x, y, row_index=1, col_index=2, channel_index=0):
        
        if self.horizontal_flip:
            if True or np.random.random() < 0.5:
                x = flip_axis(x, 2)
                y = flip_axis(y, 2)
        
        # use composition of homographies to generate final transform that needs to be applied
        if self.rotation_range:
            theta = np.pi / 180 * np.random.uniform(-self.rotation_range, self.rotation_range)
        else:
            theta = 0
        rotation_matrix = np.array([[np.cos(theta), -np.sin(theta), 0],
                                    [np.sin(theta), np.cos(theta), 0],
                                    [0, 0, 1]])
        if self.height_shift_range:
            tx = np.random.uniform(-self.height_shift_range, self.height_shift_range) * x.shape[row_index]
        else:
            tx = 0

        if self.width_shift_range:
            ty = np.random.uniform(-self.width_shift_range, self.width_shift_range) * x.shape[col_index]
        else:
            ty = 0

        translation_matrix = np.array([[1, 0, tx],
                                       [0, 1, ty],
                                       [0, 0, 1]])
        if self.shear_range:
            shear = np.random.uniform(-self.shear_range, self.shear_range)
        else:
            shear = 0
        shear_matrix = np.array([[1, -np.sin(shear), 0],
                                 [0, np.cos(shear), 0],
                                 [0, 0, 1]])

        if self.zoom_range[0] == 1 and self.zoom_range[1] == 1:
            zx, zy = 1, 1
        else:
            zx, zy = np.random.uniform(self.zoom_range[0], self.zoom_range[1], 2)
        zoom_matrix = np.array([[zx, 0, 0],
                                [0, zy, 0],
                                [0, 0, 1]])

        transform_matrix = np.dot(np.dot(np.dot(rotation_matrix, translation_matrix), shear_matrix), zoom_matrix)
        
        h, w = x.shape[row_index], x.shape[col_index]
        transform_matrix = transform_matrix_offset_center(transform_matrix, h, w)
        
        x = apply_transform(x, transform_matrix, channel_index,
                            fill_mode='constant')
        y = apply_transform(y, transform_matrix, channel_index,
                            fill_mode='constant')
        
        
        #        

        if self.vertical_flip:
            if np.random.random() < 0.5:
                x = flip_axis(x, 1)
                y = flip_axis(y, 1)
        
        if self.channel_shift_range != 0:
            x = random_channel_shift(x, self.channel_shift_range)


        if self.elastic is not None:
            x, y = elastic_transform(x.reshape(h,w), y.reshape(h,w), *self.elastic)
            x, y = x.reshape(1, h, w), y.reshape(1, h, w)
        
        return x, y
    
    def flow(self, X, Y, batch_size, shuffle=True, seed=None):
        return CustomNumpyArrayIterator(
            X, Y, self,
            batch_size=batch_size, shuffle=shuffle, seed=seed)

        
def elastic_transform(image, mask, alpha, sigma, alpha_affine=None, random_state=None):
    """Elastic deformation of images as described in [Simard2003]_ (with modifications).
    .. [Simard2003] Simard, Steinkraus and Platt, "Best Practices for
         Convolutional Neural Networks applied to Visual Document Analysis", in
         Proc. of the International Conference on Document Analysis and
         Recognition, 2003.

     Based on https://gist.github.com/erniejunior/601cdf56d2b424757de5
    """
    if random_state is None:
        random_state = np.random.RandomState(None)

    shape = image.shape

    dx = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha
    dy = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma) * alpha

    x, y = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))
    indices = np.reshape(y+dy, (-1, 1)), np.reshape(x+dx, (-1, 1))


    res_x = map_coordinates(image, indices, order=1, mode='reflect').reshape(shape)
    res_y = map_coordinates(mask, indices, order=1, mode='reflect').reshape(shape)
    return res_x, res_y


def test():
    X = np.random.randint(0,100, (1000, 1, 100, 200))
    YY = [np.random.randint(0,100, (1000, 1, 100, 200)), np.random.random((1000, 1))]
    cid = CustomImageDataGenerator(horizontal_flip=True, elastic=(100,20))
    gen = cid.flow(X, YY, batch_size=64, shuffle=False)
    n = gen.next()[0]
    
    
if __name__ == '__main__':
    sys.exit(test())


================================================
FILE: average_ensembles.py
================================================
import numpy as np
import sys
from u_model import IMG_COLS as img_cols, IMG_ROWS as img_rows
from train import Learner

ensembles = {
             'ens2': (8, 'best/ens2/res3/'), 
             'ens3': (6, 'best/ens3/res3/'), 
             'ens4': (6, 'best/ens4/res3/'), 
             'ens5': (8, 'best/ens5/res3/'), 
             'ens7': (6, 'best/ens7/res3/'), 
             'ens8': (5, 'best/ens8/res3/'),  
             }


def main():
    kfold_masks, kfold_prob = [], []
    weigths = []
    for name, (kfold, prefix) in ensembles.iteritems():
        print 'Loading name=%s, prefix=%s, kfold=%d' % (name, prefix, kfold)
        ens_x_mask = np.load(prefix + 'imgs_mask_test.npy')
        ens_x_prob = np.load(prefix + 'imgs_mask_exist_test.npy')
        kfold_masks.append(ens_x_mask)
        kfold_prob.append(ens_x_prob)
        weigths.append(kfold)
    #
    total_weight = float(sum(weigths))
    total_cnt = len(weigths)
    dlen = len(kfold_masks[0])
    res_masks = np.ndarray((dlen, 1, img_rows, img_cols), dtype=np.float32)
    res_probs = np.ndarray((dlen, ), dtype=np.float32)
    
    for i in xrange(dlen):
        masks = np.ndarray((total_cnt, 1, img_rows, img_cols), dtype=np.float32)
        probs = np.ndarray((total_cnt, ), dtype=np.float32)
        for k in xrange(total_cnt):
            masks[k] = weigths[k] * kfold_masks[k][i]
            probs[k] = weigths[k] * kfold_prob[k][i]
        res_masks[i] = np.sum(masks, 0)/total_weight
        res_probs[i] = np.sum(probs)/total_weight
    print 'Saving', Learner.test_mask_res, Learner.test_mask_exist_res
    np.save(Learner.test_mask_res, res_masks)
    np.save(Learner.test_mask_exist_res, res_probs)
    

if __name__ == '__main__':
    sys.exit(main())



================================================
FILE: current.py
================================================
import numpy as np
import sys
from data import load_test_data
from u_model import get_unet
from keras.optimizers import Adam
from train import preprocess, Learner
#aug
from u_model import IMG_COLS as img_cols, IMG_ROWS as img_rows
from keras.preprocessing.image import flip_axis, random_channel_shift

curry = lambda func, *args, **kw:\
            lambda *p, **n:\
                 func(*args + p, **dict(kw.items() + n.items()))
from keras.preprocessing.image import apply_transform, transform_matrix_offset_center


def zoom(x, zoom_range, row_index=1, col_index=2, channel_index=0,
                fill_mode='nearest', cval=0.):
    zx, zy = zoom_range
    zoom_matrix = np.array([[zx, 0, 0],
                            [0, zy, 0],
                            [0, 0, 1]])

    h, w = x.shape[row_index], x.shape[col_index]
    transform_matrix = transform_matrix_offset_center(zoom_matrix, h, w)
    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
    return x


transforms = (
              {'do': curry(flip_axis, axis=1), 'undo': curry(flip_axis, axis=1)},
              {'do': curry(flip_axis, axis=2), 'undo': curry(flip_axis, axis=2)},
              {'do': curry(zoom, zoom_range=(1.05, 1.05)), 'undo': curry(zoom, zoom_range=(1/1.05, 1/1.05))},
              {'do': curry(zoom, zoom_range=(0.95, 0.95)), 'undo': curry(zoom, zoom_range=(1/0.95, 1/0.95))},
              {'do': curry(random_channel_shift, intensity=5), 'undo': lambda x: x},
              )


def run_test():
    BS = 128
    print('Loading and preprocessing test data...')
    mean, std = Learner.load_meanstd()
    
    imgs_test = load_test_data()
    imgs_test = preprocess(imgs_test)

    imgs_test = imgs_test.astype('float32')
    imgs_test -= mean
    imgs_test /= std

    print('Loading saved weights...')
    model = get_unet(Adam(0.001))
    print ('Loading weights from %s' % Learner.best_weight_path)
    model.load_weights(Learner.best_weight_path)
    
    print ('Augment')
    alen, dlen = len(transforms), len(imgs_test)
    test_x = np.ndarray((alen, dlen, 1, img_rows, img_cols), dtype=np.float32)
    for i in xrange(dlen):
        for j, transform in enumerate(transforms):
            test_x[j,i] = transform['do'](imgs_test[i].copy())
    #
    print('Predicting masks on test data...')
    outputs = []
    asis_res = model.predict(imgs_test, batch_size=BS, verbose=1)
    outputs.append(asis_res)
    for j, transform in enumerate(transforms):
        t_y = model.predict(test_x[j], batch_size=BS, verbose=1)
        outputs.append(t_y)
    #
    print('Analyzing')
    test_masks = np.ndarray((dlen, 1, img_rows, img_cols), dtype=np.float32)
    test_probs = np.ndarray((dlen, ), dtype=np.float32)
    for i in xrange(dlen):
        masks = np.ndarray((alen+1, 1, img_rows, img_cols), dtype=np.float32)
        probs = np.ndarray((alen+1, ), dtype=np.float32)
        for j, t_y in enumerate(outputs):
            mask, prob = t_y[0][i], t_y[1][i]
            if j:
                mask = transforms[j-1]['undo'](mask)
            masks[j] = mask
            probs[j] = prob
        #
        test_masks[i] = np.mean(masks, 0)
        test_probs[i] = np.mean(probs)
            
    print('Saving ')
    np.save(Learner.test_mask_res, test_masks)
    np.save(Learner.test_mask_exist_res, test_probs)

def main():
    run_test()

if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: current_ensemble.py
================================================
import numpy as np
import sys
from data import load_test_data
from u_model import get_unet
from keras.optimizers import Adam
from train import preprocess, Learner
#aug
from u_model import IMG_COLS as img_cols, IMG_ROWS as img_rows
from keras.preprocessing.image import flip_axis, random_channel_shift

curry = lambda func, *args, **kw:\
            lambda *p, **n:\
                 func(*args + p, **dict(kw.items() + n.items()))
from keras.preprocessing.image import apply_transform, transform_matrix_offset_center


def zoom(x, zoom_range, row_index=1, col_index=2, channel_index=0,
                fill_mode='nearest', cval=0.):
    zx, zy = zoom_range
    zoom_matrix = np.array([[zx, 0, 0],
                            [0, zy, 0],
                            [0, 0, 1]])

    h, w = x.shape[row_index], x.shape[col_index]
    transform_matrix = transform_matrix_offset_center(zoom_matrix, h, w)
    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
    return x


transforms = (
              {'do': curry(flip_axis, axis=1), 'undo': curry(flip_axis, axis=1)},
              {'do': curry(flip_axis, axis=2), 'undo': curry(flip_axis, axis=2)},
              {'do': curry(zoom, zoom_range=(1.05, 1.05)), 'undo': curry(zoom, zoom_range=(1/1.05, 1/1.05))},
              {'do': curry(zoom, zoom_range=(0.95, 0.95)), 'undo': curry(zoom, zoom_range=(1/0.95, 1/0.95))},
              {'do': curry(random_channel_shift, intensity=5), 'undo': lambda x: x},
              )


def run_test():
    BS = 256
    print('Loading and preprocessing test data...')
    mean, std = Learner.load_meanstd()
    
    imgs_test = load_test_data()
#    imgs_test = imgs_test[:100]
#    print ('test')
    imgs_test = preprocess(imgs_test)

    imgs_test = imgs_test.astype('float32')
    imgs_test -= mean
    imgs_test /= std

    
    print ('Augment')
    alen, dlen = len(transforms), len(imgs_test)
    test_x = np.ndarray((alen, dlen, 1, img_rows, img_cols), dtype=np.float32)
    for i in xrange(dlen):
        for j, transform in enumerate(transforms):
            test_x[j,i] = transform['do'](imgs_test[i].copy())
    #
    kfold = 6
    kfold_masks, kfold_prob = [], []
    for _iter in xrange(kfold):
        print('Iter=%d, Loading saved weights...' % _iter)
        model = get_unet(Adam(0.001))
        filepath = Learner.best_weight_path + '_%d.fold' % _iter
        print ('Loading weights from %s' % filepath)
        model.load_weights(filepath)
        #
        print('Predicting masks on test data...')
        outputs = []
        asis_res = model.predict(imgs_test, batch_size=BS, verbose=1)
        outputs.append(asis_res)
        for j, transform in enumerate(transforms):
            t_y = model.predict(test_x[j], batch_size=BS, verbose=1)
            outputs.append(t_y)
        #
        print('Analyzing')
        test_masks = np.ndarray((dlen, 1, img_rows, img_cols), dtype=np.float32)
        test_probs = np.ndarray((dlen, ), dtype=np.float32)
        for i in xrange(dlen):
            masks = np.ndarray((alen+1, 1, img_rows, img_cols), dtype=np.float32)
            probs = np.ndarray((alen+1, ), dtype=np.float32)
            for j, t_y in enumerate(outputs):
                mask, prob = t_y[0][i], t_y[1][i]
                if j:
                    mask = transforms[j-1]['undo'](mask.copy())
                masks[j] = mask
                probs[j] = prob
            #
            test_masks[i] = np.mean(masks, 0)
            test_probs[i] = np.mean(probs)
        kfold_masks.append(test_masks)
        kfold_prob.append(test_probs)
    
    print 'Summing results of ensemble'
    #
    res_masks = np.ndarray((dlen, 1, img_rows, img_cols), dtype=np.float32)
    res_probs = np.ndarray((dlen, ), dtype=np.float32)
    for i in xrange(dlen):
        masks = np.ndarray((kfold, 1, img_rows, img_cols), dtype=np.float32)
        probs = np.ndarray((kfold, ), dtype=np.float32)
        for k in xrange(kfold):
            masks[k] = kfold_masks[k][i]
            probs[k] = kfold_prob[k][i]
        res_masks[i] = np.mean(masks, 0)
        res_probs[i] = np.mean(probs)
        

    print('Saving ')
    np.save(Learner.test_mask_res, res_masks)
    np.save(Learner.test_mask_exist_res, res_probs)


def main():
    run_test()

if __name__ == '__main__':
    sys.exit(main())



================================================
FILE: data.py
================================================
from __future__ import print_function
import os, sys
import numpy as np
import cv2

image_rows = 420
image_cols = 580

_dir = os.path.join(os.path.realpath(os.path.dirname(__file__)), '')
data_path = os.path.join(_dir, '../')
preprocess_path = os.path.join(_dir, 'np_data')
img_train_path = os.path.join(preprocess_path, 'imgs_train.npy')
img_train_mask_path = os.path.join(preprocess_path, 'imgs_mask_train.npy')
img_train_patients = os.path.join(preprocess_path, 'imgs_patient.npy')
img_test_path = os.path.join(preprocess_path, 'imgs_test.npy') 
img_test_id_path = os.path.join(preprocess_path, 'imgs_id_test.npy') 



def load_test_data():
    print ('Loading test data from %s' % img_test_path)
    imgs_test = np.load(img_test_path)
    return imgs_test

def load_test_ids():
    print ('Loading test ids from %s' % img_test_id_path)
    imgs_id = np.load(img_test_id_path)
    return imgs_id

def load_train_data():
    print ('Loading train data from %s and %s' % (img_train_path, img_train_mask_path))
    imgs_train = np.load(img_train_path)
    imgs_mask_train = np.load(img_train_mask_path)
    return imgs_train, imgs_mask_train

def load_patient_num():
    print ('Loading patient numbers from %s' % img_train_patients)
    return np.load(img_train_patients)

def get_patient_nums(string):
    pat, photo = string.split('_')
    photo = photo.split('.')[0]
    return int(pat), int(photo)

def create_train_data():
    train_data_path = os.path.join(data_path, 'train')
    images = filter((lambda image: 'mask' not in image), os.listdir(train_data_path))
    total = len(images) 

    imgs = np.ndarray((total, 1, image_rows, image_cols), dtype=np.uint8)
    imgs_mask = np.ndarray((total, 1, image_rows, image_cols), dtype=np.uint8)
    i = 0
    print('Creating training images...')
    img_patients = np.ndarray((total,), dtype=np.uint8)
    for image_name in images:
        if 'mask' in image_name:
            continue
        image_mask_name = image_name.split('.')[0] + '_mask.tif'
        patient_num = image_name.split('_')[0]
        img = cv2.imread(os.path.join(train_data_path, image_name), cv2.IMREAD_GRAYSCALE)
        img_mask = cv2.imread(os.path.join(train_data_path, image_mask_name), cv2.IMREAD_GRAYSCALE)

        imgs[i, 0] = img
        imgs_mask[i, 0] = img_mask
        img_patients[i] = patient_num
        if i % 100 == 0:
            print('Done: {0}/{1} images'.format(i, total))
        i += 1
    print('Loading done.')
    np.save(img_train_patients, img_patients)
    np.save(img_train_path, imgs)
    np.save(img_train_mask_path, imgs_mask)
    print('Saving to .npy files done.')


def create_test_data():
    train_data_path = os.path.join(data_path, 'test')
    images = os.listdir(train_data_path)
    total = len(images)

    imgs = np.ndarray((total, 1, image_rows, image_cols), dtype=np.uint8)
    imgs_id = np.ndarray((total, ), dtype=np.int32)

    i = 0
    print('Creating test images...')
    for image_name in images:
        img_id = int(image_name.split('.')[0])
        img = cv2.imread(os.path.join(train_data_path, image_name), cv2.IMREAD_GRAYSCALE)

        imgs[i, 0] = img
        imgs_id[i] = img_id

        if i % 100 == 0:
            print('Done: {0}/{1} images'.format(i, total))
        i += 1
    print('Loading done.')

    np.save(img_test_path, imgs)
    np.save(img_test_id_path, imgs_id)
    print('Saving to .npy files done.')


def main():
    create_train_data()
    create_test_data()

if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: keras_plus.py
================================================
from keras.callbacks import Callback
from keras.callbacks import warnings
import sys
import numpy as np
from keras import backend as K


class AdvancedLearnignRateScheduler(Callback):
    '''
    # Arguments
        monitor: quantity to be monitored.
        patience: number of epochs with no improvement
            after which training will be stopped.
        verbose: verbosity mode.
        mode: one of {auto, min, max}. In 'min' mode,
            training will stop when the quantity
            monitored has stopped decreasing; in 'max'
            mode it will stop when the quantity
            monitored has stopped increasing.
    '''
    def __init__(self, monitor='val_loss', patience=0,
                 verbose=0, mode='auto', decayRatio=0.5):
        super(Callback, self).__init__()

        self.monitor = monitor
        self.patience = patience
        self.verbose = verbose
        self.wait = 0
        self.decayRatio = decayRatio

        if mode not in ['auto', 'min', 'max']:
            warnings.warn('Mode %s is unknown, '
                          'fallback to auto mode.'
                          % (self.mode), RuntimeWarning)
            mode = 'auto'

        if mode == 'min':
            self.monitor_op = np.less
            self.best = np.Inf
        elif mode == 'max':
            self.monitor_op = np.greater
            self.best = -np.Inf
        else:
            if 'acc' in self.monitor:
                self.monitor_op = np.greater
                self.best = -np.Inf
            else:
                self.monitor_op = np.less
                self.best = np.Inf

    def on_epoch_end(self, epoch, logs={}):
        current = logs.get(self.monitor)

        current_lr = K.get_value(self.model.optimizer.lr)
        print(" \nLearning rate:", current_lr)
        if current is None:
            warnings.warn('AdvancedLearnignRateScheduler'
                          ' requires %s available!' %
                          (self.monitor), RuntimeWarning)

        if self.monitor_op(current, self.best):
            self.best = current
            self.wait = 0
        else:
            if self.wait >= self.patience:
                assert hasattr(self.model.optimizer, 'lr'), \
                    'Optimizer must have a "lr" attribute.'
                current_lr = K.get_value(self.model.optimizer.lr)
                new_lr = current_lr * self.decayRatio
                if self.verbose > 0:
                    print(' \nEpoch %05d: reducing learning rate' % (epoch))
                    sys.stderr.write(' \nnew lr: %.5f\n' % new_lr)
                K.set_value(self.model.optimizer.lr, new_lr)
                self.wait = 0

            self.wait += 1


class LearningRateDecay(Callback):
    '''Learning rate scheduler.

    # Arguments
        schedule: a function that takes an epoch index as input
            (integer, indexed from 0) and returns a new
            learning rate as output (float).
    '''
    def __init__(self, decay, every_n=1, verbose=0):
        Callback.__init__(self)
        self.decay = decay
        self.every_n = every_n
        self.verbose = verbose

    def on_epoch_end(self, epoch, logs={}):
        if not (epoch and epoch % self.every_n == 0):
            return

        assert hasattr(self.model.optimizer, 'lr'), \
            'Optimizer must have a "lr" attribute.'
        current_lr = K.get_value(self.model.optimizer.lr)
        new_lr = current_lr * self.decay
        if self.verbose > 0:
            print(' \nEpoch %05d: reducing learning rate' % (epoch))
            sys.stderr.write('new lr: %.5f\n' % new_lr)
        K.set_value(self.model.optimizer.lr, new_lr)


================================================
FILE: metric.py
================================================
import sys
import numpy as np
from keras import backend as K
smooth = 1


def mean_length_error(y_true, y_pred):
    y_true_f = K.sum(K.round(K.flatten(y_true)))
    y_pred_f = K.sum(K.round(K.flatten(y_pred)))
    delta = (y_pred_f - y_true_f)
    return K.mean(K.tanh(delta))

def dice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

def dice_coef_loss(y_true, y_pred):
    return -dice_coef(y_true, y_pred)

def np_dice_coef(y_true, y_pred):
    tr = y_true.flatten()
    pr = y_pred.flatten()
    return (2. * np.sum(tr * pr) + smooth) / (np.sum(tr) + np.sum(pr) + smooth)


def main():
    a = np.random.random((420,100))
    b = np.random.random((420,100))
#    print a.flatten().shape
    res =  np_dice_coef(a,b )
    print res


if __name__ == '__main__':
    sys.exit(main())
    

================================================
FILE: submission.py
================================================
from __future__ import print_function
import sys, os
import numpy as np
import cv2
from data import image_cols, image_rows, load_test_ids
from train import Learner


def prep(img):
    img = img.astype('float32')
    img = cv2.resize(img, (image_cols, image_rows)) 
    img = cv2.threshold(img, 0.5, 1., cv2.THRESH_BINARY)[1].astype(np.uint8)
    return img

def run_length_enc(label):
    from itertools import chain
    x = label.transpose().flatten()
    y = np.where(x > 0)[0]
    if len(y) < 10:  # consider as empty
        return ''
    z = np.where(np.diff(y) > 1)[0]
    start = np.insert(y[z+1], 0, y[0])
    end = np.append(y[z], y[-1])
    length = end - start
    res = [[s+1, l+1] for s, l in zip(list(start), list(length))]
    res = list(chain.from_iterable(res))
    return ' '.join([str(r) for r in res])


def submission():
    imgs_id_test = load_test_ids()
    
    print ('Loading test_mask_res from %s' % Learner.test_mask_res)
    imgs_test = np.load(Learner.test_mask_res)
    print ('Loading imgs_exist_test from %s' % Learner.test_mask_exist_res)
    imgs_exist_test = np.load(Learner.test_mask_exist_res)

    argsort = np.argsort(imgs_id_test)
    imgs_id_test = imgs_id_test[argsort]
    imgs_test = imgs_test[argsort]
    imgs_exist_test = imgs_exist_test[argsort]

    total = imgs_test.shape[0]
    ids = []
    rles = []
    for i in xrange(total):
        img = imgs_test[i, 0]
        img_exist = imgs_exist_test[i]
        img = prep(img)
        new_prob = (img_exist + min(1, np.sum(img)/10000.0 )* 5 / 3)/2
        if np.sum(img) > 0 and new_prob < 0.5:
            img = np.zeros((image_rows, image_cols))

        rle = run_length_enc(img)

        rles.append(rle)
        ids.append(imgs_id_test[i])

        if i % 1000 == 0:
            print('{}/{}'.format(i, total))

    file_name = os.path.join(Learner.res_dir, 'submission.csv')

    with open(file_name, 'w+') as f:
        f.write('img,pixels\n')
        for i in xrange(total):
            s = str(ids[i]) + ',' + rles[i]
            f.write(s + '\n')

def main():
    submission()


if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: train.py
================================================
from __future__ import print_function
from optparse import OptionParser
import cv2, sys, os, shutil, random
import numpy as np
from keras.optimizers import Adam
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras.preprocessing.image import flip_axis, random_channel_shift
from keras.engine.training import slice_X
from keras_plus import LearningRateDecay
from u_model import get_unet, IMG_COLS as img_cols, IMG_ROWS as img_rows
from data import load_train_data, load_test_data, load_patient_num
from augmentation import random_zoom, elastic_transform, random_rotation
from utils import save_pickle, load_pickle, count_enum

_dir = os.path.join(os.path.realpath(os.path.dirname(__file__)), '')


def preprocess(imgs, to_rows=None, to_cols=None):
    if to_rows is None or to_cols is None:
        to_rows = img_rows
        to_cols = img_cols
    imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], to_rows, to_cols), dtype=np.uint8)
    for i in xrange(imgs.shape[0]):
        imgs_p[i, 0] = cv2.resize(imgs[i, 0], (to_cols, to_rows), interpolation=cv2.INTER_CUBIC)
    return imgs_p


class Learner(object):
    
    suffix = ''
    res_dir = os.path.join(_dir, 'res' + suffix)
    best_weight_path = os.path.join(res_dir, 'unet.hdf5')
    test_mask_res = os.path.join(res_dir, 'imgs_mask_test.npy')
    test_mask_exist_res = os.path.join(res_dir, 'imgs_mask_exist_test.npy')
    meanstd_path = os.path.join(res_dir, 'meanstd.dump')
    valid_data_path = os.path.join(res_dir, 'valid.npy')
    tensorboard_dir = os.path.join(res_dir, 'tb')
    
    def __init__(self, model_func, validation_split):
        self.model_func = model_func
        self.validation_split = validation_split
        self.__iter_res_dir = os.path.join(self.res_dir, 'res_iter')
        self.__iter_res_file = os.path.join(self.__iter_res_dir, '{epoch:02d}-{val_loss:.4f}.unet.hdf5')
        
    def _dir_init(self):
        if not os.path.exists(self.res_dir):
            os.mkdir(self.res_dir)
        #iter clean
        if os.path.exists(self.__iter_res_dir):
            shutil.rmtree(self.__iter_res_dir)
        os.mkdir(self.__iter_res_dir)
    
    def save_meanstd(self):
        data = [self.mean, self.std]
        save_pickle(self.meanstd_path, data)
        
    @classmethod
    def load_meanstd(cls):
        print ('Load meanstd from %s' % cls.meanstd_path)
        mean, std = load_pickle(cls.meanstd_path)
        return mean, std
    
    @classmethod
    def save_valid_idx(cls, idx):
        save_pickle(cls.valid_data_path, idx)
        
    @classmethod
    def load_valid_idx(cls):
        return load_pickle(cls.valid_data_path)
    
    def _init_mean_std(self, data):
        data = data.astype('float32')
        self.mean, self.std = np.mean(data), np.std(data)
        self.save_meanstd()
        return data
    
    def get_object_existance(self, mask_array):
        return np.array([int(np.sum(mask_array[i, 0]) > 0) for i in xrange(len(mask_array))])

    def standartize(self, array, to_float=False):
        if to_float:
            array = array.astype('float32')
        if self.mean is None or self.std is None:
            raise ValueError, 'No mean/std is initialised'
        
        array -= self.mean
        array /= self.std
        return array

    @classmethod
    def norm_mask(cls, mask_array):
        mask_array = mask_array.astype('float32')
        mask_array /= 255.0
        return mask_array

    @classmethod
    def shuffle_train(cls, data, mask):
        perm = np.random.permutation(len(data))
        data = data[perm]
        mask = mask[perm]
        return data, mask

    @classmethod
    def split_train_and_valid_by_patient(cls, data, mask, validation_split, shuffle=False):
        print('Shuffle & split...')
        patient_nums = load_patient_num()
        patient_dict = count_enum(patient_nums)
        pnum = len(patient_dict)
        val_num = int(pnum * validation_split)
        patients = patient_dict.keys()
        if shuffle:
            random.shuffle(patients)
        val_p, train_p = patients[:val_num], patients[val_num:]
        train_indexes = [i for i, c in enumerate(patient_nums) if c in set(train_p)]
        val_indexes = [i for i, c in enumerate(patient_nums) if c in set(val_p)]
        x_train, y_train = data[train_indexes], mask[train_indexes]
        x_valid, y_valid = data[val_indexes], mask[val_indexes]
        cls.save_valid_idx(val_indexes)
        print ('val patients:', len(x_valid), val_p)
        print ('train patients:', len(x_train), train_p)
        return (x_train, y_train), (x_valid, y_valid)

    @classmethod
    def split_train_and_valid(cls, data, mask, validation_split, shuffle=False):
        print('Shuffle & split...')
        if shuffle:
            data, mask = cls.shuffle_train(data, mask)
        split_at = int(len(data) * (1. - validation_split))
        x_train, x_valid = (slice_X(data, 0, split_at), slice_X(data, split_at))
        y_train, y_valid = (slice_X(mask, 0, split_at), slice_X(mask, split_at))
        cls.save_valid_idx(range(len(data))[split_at:])
        return (x_train, y_train), (x_valid, y_valid)
        
    def test(self, model, batch_size=256):
        print('Loading and pre-processing test data...')
        imgs_test = load_test_data()
        imgs_test = preprocess(imgs_test)
        imgs_test = self.standartize(imgs_test, to_float=True)
    
        print('Loading best saved weights...')
        model.load_weights(self.best_weight_path)
        print('Predicting masks on test data and saving...')
        imgs_mask_test = model.predict(imgs_test, batch_size=batch_size, verbose=1)
        
        np.save(self.test_mask_res, imgs_mask_test[0])
        np.save(self.test_mask_exist_res, imgs_mask_test[1])
        
    def __pretrain_model_load(self, model, pretrained_path):
        if pretrained_path is not None:
            if not os.path.exists(pretrained_path):
                raise ValueError, 'No such pre-trained path exists'
            model.load_weights(pretrained_path)
            
            
    def augmentation(self, X, Y):
        print('Augmentation model...')
        total = len(X)
        x_train, y_train = [], []
        
        for i in xrange(total):
            x, y = X[i], Y[i]
            #standart
            x_train.append(x)
            y_train.append(y)
        
#            for _ in xrange(1):
#                _x, _y = elastic_transform(x[0], y[0], 100, 20)
#                x_train.append(_x.reshape((1,) + _x.shape))
#                y_train.append(_y.reshape((1,) + _y.shape))
            
            #flip x
            x_train.append(flip_axis(x, 2))
            y_train.append(flip_axis(y, 2))
            #flip y
            x_train.append(flip_axis(x, 1))
            y_train.append(flip_axis(y, 1))
            #continue
            #zoom
            for _ in xrange(1):
                _x, _y = random_zoom(x, y, (0.9, 1.1))
                x_train.append(_x)
                y_train.append(_y)
            for _ in xrange(0):
                _x, _y = random_rotation(x, y, 5)
                x_train.append(_x)
                y_train.append(_y)
            #intentsity
            for _ in xrange(1):
                _x = random_channel_shift(x, 5.0)
                x_train.append(_x)
                y_train.append(y)
    
        x_train = np.array(x_train)
        y_train = np.array(y_train)
        return x_train, y_train
        
    def fit(self, x_train, y_train, x_valid, y_valid, pretrained_path):
        print('Creating and compiling and fitting model...')
        print('Shape:', x_train.shape)
        #second output
        y_train_2 = self.get_object_existance(y_train)
        y_valid_2 = self.get_object_existance(y_valid)

        #load model
        optimizer = Adam(lr=0.0045)
        model = self.model_func(optimizer)

        #checkpoints
        model_checkpoint = ModelCheckpoint(self.__iter_res_file, monitor='val_loss')
        model_save_best = ModelCheckpoint(self.best_weight_path, monitor='val_loss', save_best_only=True)
        early_s = EarlyStopping(monitor='val_loss', patience=5, verbose=1)
        learning_rate_adapt = LearningRateDecay(0.9, every_n=2, verbose=1)
        self.__pretrain_model_load(model, pretrained_path)
        model.fit(
                   x_train, [y_train, y_train_2], 
                   validation_data=(x_valid, [y_valid, y_valid_2]),
                   batch_size=128, nb_epoch=50,
                   verbose=1, shuffle=True,
                   callbacks=[model_save_best, model_checkpoint, early_s]
                   ) 
        
        #augment
        return model

    def train_and_predict(self, pretrained_path=None, split_random=True):
        self._dir_init()
        print('Loading and preprocessing and standarize train data...')
        imgs_train, imgs_mask_train = load_train_data()
        
        imgs_train = preprocess(imgs_train)

        imgs_mask_train = preprocess(imgs_mask_train)
        
        imgs_mask_train = self.norm_mask(imgs_mask_train)

        split_func = split_random and self.split_train_and_valid or self.split_train_and_valid_by_patient
        (x_train, y_train), (x_valid, y_valid) = split_func(imgs_train, imgs_mask_train,
                                                        validation_split=self.validation_split)
        self._init_mean_std(x_train)
        x_train = self.standartize(x_train, True)
        x_valid = self.standartize(x_valid, True)
        #augmentation
        x_train, y_train = self.augmentation(x_train, y_train)
        #fit
        model = self.fit(x_train, y_train, x_valid, y_valid, pretrained_path)
        #test
        self.test(model)


def main():
    parser = OptionParser()
    parser.add_option("-s", "--split_random", action='store', type='int', dest='split_random', default = 1)
    parser.add_option("-m", "--model_name", action='store', type='str', dest='model_name', default = 'u_model')
    #
    options, _ = parser.parse_args()
    split_random = options.split_random
    model_name = options.model_name
    if model_name is None:
        raise ValueError, 'model_name is not defined'
    #
    import imp
    model_ = imp.load_source('model_', model_name + '.py')
    model_func = model_.get_unet
    #
    lr = Learner(model_func, validation_split=0.2)
    lr.train_and_predict(pretrained_path=None, split_random=split_random)
    print ('Results in ', lr.res_dir)

if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: train_generator.py
================================================
from __future__ import print_function
from optparse import OptionParser
import cv2, sys, os, shutil, random
import numpy as np
from keras.optimizers import Adam, SGD, RMSprop
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras.preprocessing.image import flip_axis, random_channel_shift
from keras.engine.training import slice_X
from keras_plus import LearningRateDecay
from u_model import get_unet, IMG_COLS as img_cols, IMG_ROWS as img_rows
from data import load_train_data, load_test_data, load_patient_num
from augmentation import CustomImageDataGenerator
from augmentation import random_zoom, elastic_transform, load_aug
from utils import save_pickle, load_pickle, count_enum

_dir = os.path.join(os.path.realpath(os.path.dirname(__file__)), '')



def preprocess(imgs, to_rows=None, to_cols=None):
    if to_rows is None or to_cols is None:
        to_rows = img_rows
        to_cols = img_cols
    imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], to_rows, to_cols), dtype=np.uint8)
    for i in xrange(imgs.shape[0]):
        imgs_p[i, 0] = cv2.resize(imgs[i, 0], (to_cols, to_rows), interpolation=cv2.INTER_CUBIC)
    return imgs_p

class Learner(object):
    
    suffix = ''
    res_dir = os.path.join(_dir, 'res' + suffix)
    best_weight_path = os.path.join(res_dir, 'unet.hdf5')
    test_mask_res = os.path.join(res_dir, 'imgs_mask_test.npy')
    test_mask_exist_res = os.path.join(res_dir, 'imgs_mask_exist_test.npy')
    meanstd_path = os.path.join(res_dir, 'meanstd.dump')
    valid_data_path = os.path.join(res_dir, 'valid.npy')
    tensorboard_dir = os.path.join(res_dir, 'tb')
    
    def __init__(self, model_func, validation_split):
        self.model_func = model_func
        self.validation_split = validation_split
        self.__iter_res_dir = os.path.join(self.res_dir, 'res_iter')
        self.__iter_res_file = os.path.join(self.__iter_res_dir, '{epoch:02d}-{val_loss:.4f}.unet.hdf5')
        
    def _dir_init(self):
        if not os.path.exists(self.res_dir):
            os.mkdir(self.res_dir)
        #iter clean
        if os.path.exists(self.__iter_res_dir):
            shutil.rmtree(self.__iter_res_dir)
        os.mkdir(self.__iter_res_dir)
    
    def save_meanstd(self):
        data = [self.mean, self.std]
        save_pickle(self.meanstd_path, data)
        
    @classmethod
    def load_meanstd(cls):
        print ('Load meanstd from %s' % cls.meanstd_path)
        mean, std = load_pickle(cls.meanstd_path)
        return mean, std
    
    @classmethod
    def save_valid_idx(cls, idx):
        save_pickle(cls.valid_data_path, idx)
        
    @classmethod
    def load_valid_idx(cls):
        return load_pickle(cls.valid_data_path)
    
    def _init_mean_std(self, data):
        data = data.astype('float32')
        self.mean, self.std = np.mean(data), np.std(data)
        self.save_meanstd()
        return data
    
    def get_object_existance(self, mask_array):
        return np.array([int(np.sum(mask_array[i, 0]) > 0) for i in xrange(len(mask_array))])

    def standartize(self, array, to_float=False):
        if to_float:
            array = array.astype('float32')
        if self.mean is None or self.std is None:
            raise ValueError, 'No mean/std is initialised'
        
        array -= self.mean
        array /= self.std
        return array

    @classmethod
    def norm_mask(cls, mask_array):
        mask_array = mask_array.astype('float32')
        mask_array /= 255.0
        return mask_array

    @classmethod
    def shuffle_train(cls, data, mask):
        perm = np.random.permutation(len(data))
        data = data[perm]
        mask = mask[perm]
        return data, mask

    @classmethod
    def split_train_and_valid_by_patient(cls, data, mask, validation_split, shuffle=False):
        print('Shuffle & split...')
        patient_nums = load_patient_num()
        patient_dict = count_enum(patient_nums)
        pnum = len(patient_dict)
        val_num = int(pnum * validation_split)
        patients = patient_dict.keys()
        if shuffle:
            random.shuffle(patients)
        val_p, train_p = patients[:val_num], patients[val_num:]
        train_indexes = [i for i, c in enumerate(patient_nums) if c in set(train_p)]
        val_indexes = [i for i, c in enumerate(patient_nums) if c in set(val_p)]
        x_train, y_train = data[train_indexes], mask[train_indexes]
        x_valid, y_valid = data[val_indexes], mask[val_indexes]
        cls.save_valid_idx(val_indexes)
        print ('val patients:', len(x_valid), val_p)
        print ('train patients:', len(x_train), train_p)
        return (x_train, y_train), (x_valid, y_valid)

    @classmethod
    def split_train_and_valid(cls, data, mask, validation_split, shuffle=False):
        print('Shuffle & split...')
        if shuffle:
            data, mask = cls.shuffle_train(data, mask)
        split_at = int(len(data) * (1. - validation_split))
        x_train, x_valid = (slice_X(data, 0, split_at), slice_X(data, split_at))
        y_train, y_valid = (slice_X(mask, 0, split_at), slice_X(mask, split_at))
        cls.save_valid_idx(range(len(data))[split_at:])
        return (x_train, y_train), (x_valid, y_valid)
        
    def test(self, model, batch_size=256):
        print('Loading and pre-processing test data...')
        imgs_test = load_test_data()
        imgs_test = preprocess(imgs_test)
        imgs_test = self.standartize(imgs_test, to_float=True)
    
        print('Loading best saved weights...')
        model.load_weights(self.best_weight_path)
        print('Predicting masks on test data and saving...')
        imgs_mask_test = model.predict(imgs_test, batch_size=batch_size, verbose=1)
        
        np.save(self.test_mask_res, imgs_mask_test[0])
        np.save(self.test_mask_exist_res, imgs_mask_test[1])
        
    def __pretrain_model_load(self, model, pretrained_path):
        if pretrained_path is not None:
            if not os.path.exists(pretrained_path):
                raise ValueError, 'No such pre-trained path exists'
            model.load_weights(pretrained_path)
            
            
    def augmentation(self, X, Y):
        print('Augmentation model...')
        total = len(X)
        x_train, y_train = [], []
        
        for i in xrange(total):
            if i % 100 == 0:
                print ('Aug', i)
            x, y = X[i], Y[i]
            #standart
            x_train.append(x)
            y_train.append(y)
        
            for _ in xrange(2):
                _x, _y = elastic_transform(x[0], y[0], 100, 20)
                x_train.append(_x.reshape((1,) + _x.shape))
                y_train.append(_y.reshape((1,) + _y.shape))
            
            #flip x
            x_train.append(flip_axis(x, 2))
            y_train.append(flip_axis(y, 2))
            #flip y
            x_train.append(flip_axis(x, 1))
            y_train.append(flip_axis(y, 1))
            continue
            #zoom
            for _ in xrange(1):
                _x, _y = random_zoom(x, y, (0.9, 1.1))
                x_train.append(_x)
                y_train.append(_y)
            #intentsity
            for _ in xrange(1):
                _x = random_channel_shift(x, 5.0)
                x_train.append(_x)
                y_train.append(y)
                
#        for j in xrange(5):
#            xs, ys = load_aug(j)
#            ys = self.norm_mask(ys)
#            (xn, yn), _ = self.split_train_and_valid_by_patient(xs, ys, validation_split=self.validation_split, shuffle=False)
#            for i in xrange(len(xn)):
#                x_train.append(xn[i])
#                y_train.append(yn[i])
    
        x_train = np.array(x_train)
        y_train = np.array(y_train)
        return x_train, y_train
        
    def fit(self, x_train, y_train, x_valid, y_valid, pretrained_path):
        print('Creating and compiling and fitting model...')
        print('Shape:', x_train.shape)
        #second output
        y_train_2 = self.get_object_existance(y_train)
        y_valid_2 = self.get_object_existance(y_valid)

        #load model
        optimizer = Adam(lr=0.0045)
        #model = get_unet(optimizer)
        model = self.model_func(optimizer)

        #checkpoints
        model_checkpoint = ModelCheckpoint(self.__iter_res_file, monitor='val_loss')
        model_save_best = ModelCheckpoint(self.best_weight_path, monitor='val_loss', save_best_only=True)
        early_s = EarlyStopping(monitor='val_loss', patience=10, verbose=1)
        #tb = TensorBoard(self.tensorboard_dir, histogram_freq=2, write_graph=True)
        #learning_rate_adapt = AdvancedLearnignRateScheduler(monitor='val_loss', patience=1, verbose=1, mode='min', decayRatio=0.5)
        learning_rate_adapt = LearningRateDecay(0.95, every_n=4, verbose=1)
        self.__pretrain_model_load(model, pretrained_path)
        #augment
        datagen = CustomImageDataGenerator(zoom_range=(0.9,1.1),
                                           horizontal_flip=True,
                                           vertical_flip=False, 
#                                           rotation_range=5,
                                           channel_shift_range=5.0,
                                           elastic=None #(100, 20)
                                           )
#        #fit
        model.fit_generator(datagen.flow(x_train, [y_train, y_train_2], batch_size=64),
                            samples_per_epoch=len(x_train),
                            nb_epoch=250,
                            verbose=1,
                            callbacks=[model_save_best, model_checkpoint, early_s, learning_rate_adapt],
                            validation_data=(x_valid, [y_valid, y_valid_2])
                            )
        return model

    def train_and_predict(self, pretrained_path=None, split_random=True):
        self._dir_init()
        print('Loading and preprocessing and standarize train data...')
        imgs_train, imgs_mask_train = load_train_data()
        
        imgs_train = preprocess(imgs_train)

        imgs_mask_train = preprocess(imgs_mask_train)
        
        imgs_mask_train = self.norm_mask(imgs_mask_train)
        #imgs_train = self.norm_mask(imgs_train) /255
        #shuffle and split
        split_func = split_random and self.split_train_and_valid or self.split_train_and_valid_by_patient
        (x_train, y_train), (x_valid, y_valid) = split_func(imgs_train, imgs_mask_train,
                                                        validation_split=self.validation_split)
        self._init_mean_std(x_train)
        x_train = self.standartize(x_train, True)
        x_valid = self.standartize(x_valid, True)
        #fit
        model = self.fit(x_train, y_train, x_valid, y_valid, pretrained_path)
        #test
        self.test(model)


def main():
    parser = OptionParser()
    parser.add_option("-m", "--model_name", action='store', type='str', dest='model_name', default = 'u_model')
    parser.add_option("-s", "--split_random", action='store', type='int', dest='split_random', default = 1)
    #
    options, _ = parser.parse_args()
    model_name = options.model_name
    split_random = options.split_random
    if model_name is None:
        raise ValueError, 'model_name is not defined'
    #
    import imp
    model_ = imp.load_source('model_', model_name + '.py')
    model_func = model_.get_unet
    #
    lr = Learner(model_func, validation_split=0.2)
    lr.train_and_predict(pretrained_path=None, split_random=split_random)
    print ('Results in ', lr.res_dir)

if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: train_kfold.py
================================================
from optparse import OptionParser
import cv2, sys, os, shutil, random
import numpy as np
from keras.optimizers import Adam, SGD, RMSprop
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras.preprocessing.image import flip_axis, random_channel_shift
from keras.engine.training import slice_X
from keras_plus import LearningRateDecay
from u_model import get_unet, IMG_COLS as img_cols, IMG_ROWS as img_rows
from data import load_train_data, load_test_data, load_patient_num
from augmentation import CustomImageDataGenerator
from augmentation import random_zoom, elastic_transform, random_rotation
from utils import save_pickle, load_pickle, count_enum
from sklearn.cross_validation import KFold

_dir = os.path.join(os.path.realpath(os.path.dirname(__file__)), '')



def preprocess(imgs, to_rows=None, to_cols=None):
    if to_rows is None or to_cols is None:
        to_rows = img_rows
        to_cols = img_cols
    imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], to_rows, to_cols), dtype=np.uint8)
    for i in xrange(imgs.shape[0]):
        imgs_p[i, 0] = cv2.resize(imgs[i, 0], (to_cols, to_rows), interpolation=cv2.INTER_CUBIC)
    return imgs_p

class Learner(object):
    
    suffix = ''
    res_dir = os.path.join(_dir, 'res' + suffix)
    best_weight_path = os.path.join(res_dir, 'unet.hdf5')
    test_mask_res = os.path.join(res_dir, 'imgs_mask_test.npy')
    test_mask_exist_res = os.path.join(res_dir, 'imgs_mask_exist_test.npy')
    meanstd_path = os.path.join(res_dir, 'meanstd.dump')
    valid_data_path = os.path.join(res_dir, 'valid.npy')
    tensorboard_dir = os.path.join(res_dir, 'tb')
    
    def __init__(self, model_func, validation_split):
        self.model_func = model_func
        self.validation_split = validation_split
        self.__iter_res_dir = os.path.join(self.res_dir, 'res_iter')
        self.__iter_res_file = os.path.join(self.__iter_res_dir, '{epoch:02d}-{val_loss:.4f}.unet.hdf5')
        
    def _dir_init(self):
        if not os.path.exists(self.res_dir):
            os.mkdir(self.res_dir)
        #iter clean
        if os.path.exists(self.__iter_res_dir):
            shutil.rmtree(self.__iter_res_dir)
        os.mkdir(self.__iter_res_dir)
    
    def save_meanstd(self):
        data = [self.mean, self.std]
        save_pickle(self.meanstd_path, data)
        
    @classmethod
    def load_meanstd(cls):
        print ('Load meanstd from %s' % cls.meanstd_path)
        mean, std = load_pickle(cls.meanstd_path)
        return mean, std
    
    @classmethod
    def save_valid_idx(cls, idx):
        save_pickle(cls.valid_data_path, idx)
        
    @classmethod
    def load_valid_idx(cls):
        return load_pickle(cls.valid_data_path)
    
    def _init_mean_std(self, data):
        data = data.astype('float32')
        self.mean, self.std = np.mean(data), np.std(data)
        self.save_meanstd()
        return data
    
    def get_object_existance(self, mask_array):
        return np.array([int(np.sum(mask_array[i, 0]) > 0) for i in xrange(len(mask_array))])

    def standartize(self, array, to_float=False):
        if to_float:
            array = array.astype('float32')
        if self.mean is None or self.std is None:
            raise ValueError, 'No mean/std is initialised'
        
        array -= self.mean
        array /= self.std
        return array

    @classmethod
    def norm_mask(cls, mask_array):
        mask_array = mask_array.astype('float32')
        mask_array /= 255.0
        return mask_array

    @classmethod
    def shuffle_train(cls, data, mask):
        perm = np.random.permutation(len(data))
        data = data[perm]
        mask = mask[perm]
        return data, mask
        
    def __pretrain_model_load(self, model, pretrained_path):
        if pretrained_path is not None:
            if not os.path.exists(pretrained_path):
                raise ValueError, 'No such pre-trained path exists'
            model.load_weights(pretrained_path)
            
            
    def augmentation(self, X, Y):
        print('Augmentation model...')
        total = len(X)
        x_train, y_train = [], []
        
        for i in xrange(total):
            if i % 100 == 0:
                print ('Aug', i)
            x, y = X[i], Y[i]
            #standart
            x_train.append(x)
            y_train.append(y)
        
#            for _ in xrange(1):
#                _x, _y = elastic_transform(x[0], y[0], 100, 20)
#                x_train.append(_x.reshape((1,) + _x.shape))
#                y_train.append(_y.reshape((1,) + _y.shape))
            
            #flip x
            x_train.append(flip_axis(x, 2))
            y_train.append(flip_axis(y, 2))
            #flip y
            x_train.append(flip_axis(x, 1))
            y_train.append(flip_axis(y, 1))
            #continue
            #zoom
            for _ in xrange(1):
                _x, _y = random_zoom(x, y, (0.9, 1.1))
                x_train.append(_x)
                y_train.append(_y)
            for _ in xrange(0):
                _x, _y = random_rotation(x, y, 5)
                x_train.append(_x)
                y_train.append(_y)
            #intentsity
            for _ in xrange(1):
                _x = random_channel_shift(x, 5.0)
                x_train.append(_x)
                y_train.append(y)
    
        x_train = np.array(x_train)
        y_train = np.array(y_train)
        return x_train, y_train
        
    def fit(self, x_train, y_train, nfolds=8):
        print('Creating and compiling and fitting model...')
        print('Shape:', x_train.shape)
        random_state = 51
        kf = KFold(len(x_train), n_folds=nfolds, shuffle=True, random_state=random_state)
        for i, (train_index, test_index) in enumerate(kf):
            print 'Fold %d' % i
            X_train, X_valid = x_train[train_index], x_train[test_index]
            Y_train, Y_valid = y_train[train_index], y_train[test_index]
            Y_valid_2 = self.get_object_existance(Y_valid)
            X_train, Y_train = self.augmentation(X_train, Y_train)
            Y_train_2 = self.get_object_existance(Y_train)
            #
            optimizer = Adam(lr=0.0045)
            model = self.model_func(optimizer)
            model_checkpoint = ModelCheckpoint(self.__iter_res_file + '_%d.fold' % i, monitor='val_loss')
            model_save_best = ModelCheckpoint(self.best_weight_path + '_%d.fold' % i, monitor='val_loss',
                                               save_best_only=True)
            early_s = EarlyStopping(monitor='val_loss', patience=8, verbose=1)
            #
            model.fit(
                       X_train, [Y_train, Y_train_2], 
                       validation_data=(X_valid, [Y_valid, Y_valid_2]),
                       batch_size=128, nb_epoch=40,
                       verbose=1, shuffle=True,
                       callbacks=[model_save_best, model_checkpoint, early_s]
                       ) 
        
        #augment
        return model

    def train_and_predict(self, pretrained_path=None):
        self._dir_init()
        print('Loading and preprocessing and standarize train data...')
        imgs_train, imgs_mask_train = load_train_data()
        imgs_train = preprocess(imgs_train)
        imgs_mask_train = preprocess(imgs_mask_train)
        imgs_mask_train = self.norm_mask(imgs_mask_train)
        
        self._init_mean_std(imgs_train)
        imgs_train = self.standartize(imgs_train, True)
        self.fit(imgs_train, imgs_mask_train)


def main():
    parser = OptionParser()
    parser.add_option("-s", "--suffix", action='store', type='str', dest='suffix', default = None)
    parser.add_option("-m", "--model_name", action='store', type='str', dest='model_name', default = 'u_model')
    #
    options, _ = parser.parse_args()
    suffix = options.suffix
    model_name = options.model_name
    if model_name is None:
        raise ValueError, 'model_name is not defined'
#    if suffix is None:
#        raise ValueError, 'Please specify suffix option'
#    print ('Suffix: "%s"' % suffix )
    #
    import imp
    model_ = imp.load_source('model_', model_name + '.py')
    model_func = model_.get_unet
    #
    lr = Learner(model_func, validation_split=0.2)
    lr.train_and_predict(pretrained_path=None)
    print ('Results in ', lr.res_dir)

if __name__ == '__main__':
    sys.exit(main())


================================================
FILE: u_model.py
================================================
import sys
from keras.models import Model
from keras.layers import Input, merge, Convolution2D, MaxPooling2D, UpSampling2D, Dense
from keras.layers import BatchNormalization, Dropout, Flatten, Lambda
from keras.layers.advanced_activations import ELU, LeakyReLU
from metric import dice_coef, dice_coef_loss

IMG_ROWS, IMG_COLS = 80, 112 

def _shortcut(_input, residual):
    stride_width = _input._keras_shape[2] / residual._keras_shape[2]
    stride_height = _input._keras_shape[3] / residual._keras_shape[3]
    equal_channels = residual._keras_shape[1] == _input._keras_shape[1]

    shortcut = _input
    # 1 X 1 conv if shape is different. Else identity.
    if stride_width > 1 or stride_height > 1 or not equal_channels:
        shortcut = Convolution2D(nb_filter=residual._keras_shape[1], nb_row=1, nb_col=1,
                                 subsample=(stride_width, stride_height),
                                 init="he_normal", border_mode="valid")(_input)

    return merge([shortcut, residual], mode="sum")


def inception_block(inputs, depth, batch_mode=0, splitted=False, activation='relu'):
    assert depth % 16 == 0
    actv = activation == 'relu' and (lambda: LeakyReLU(0.0)) or activation == 'elu' and (lambda: ELU(1.0)) or None
    
    c1_1 = Convolution2D(depth/4, 1, 1, init='he_normal', border_mode='same')(inputs)
    
    c2_1 = Convolution2D(depth/8*3, 1, 1, init='he_normal', border_mode='same')(inputs)
    c2_1 = actv()(c2_1)
    if splitted:
        c2_2 = Convolution2D(depth/2, 1, 3, init='he_normal', border_mode='same')(c2_1)
        c2_2 = BatchNormalization(mode=batch_mode, axis=1)(c2_2)
        c2_2 = actv()(c2_2)
        c2_3 = Convolution2D(depth/2, 3, 1, init='he_normal', border_mode='same')(c2_2)
    else:
        c2_3 = Convolution2D(depth/2, 3, 3, init='he_normal', border_mode='same')(c2_1)
    
    c3_1 = Convolution2D(depth/16, 1, 1, init='he_normal', border_mode='same')(inputs)
    #missed batch norm
    c3_1 = actv()(c3_1)
    if splitted:
        c3_2 = Convolution2D(depth/8, 1, 5, init='he_normal', border_mode='same')(c3_1)
        c3_2 = BatchNormalization(mode=batch_mode, axis=1)(c3_2)
        c3_2 = actv()(c3_2)
        c3_3 = Convolution2D(depth/8, 5, 1, init='he_normal', border_mode='same')(c3_2)
    else:
        c3_3 = Convolution2D(depth/8, 5, 5, init='he_normal', border_mode='same')(c3_1)
    
    p4_1 = MaxPooling2D(pool_size=(3,3), strides=(1,1), border_mode='same')(inputs)
    c4_2 = Convolution2D(depth/8, 1, 1, init='he_normal', border_mode='same')(p4_1)
    
    res = merge([c1_1, c2_3, c3_3, c4_2], mode='concat', concat_axis=1)
    res = BatchNormalization(mode=batch_mode, axis=1)(res)
    res = actv()(res)
    return res
    

def rblock(inputs, num, depth, scale=0.1):    
    residual = Convolution2D(depth, num, num, border_mode='same')(inputs)
    residual = BatchNormalization(mode=2, axis=1)(residual)
    residual = Lambda(lambda x: x*scale)(residual)
    res = _shortcut(inputs, residual)
    return ELU()(res) 
    

def NConvolution2D(nb_filter, nb_row, nb_col, border_mode='same', subsample=(1, 1)):
    def f(_input):
        conv = Convolution2D(nb_filter=nb_filter, nb_row=nb_row, nb_col=nb_col, subsample=subsample,
                              border_mode=border_mode)(_input)
        norm = BatchNormalization(mode=2, axis=1)(conv)
        return ELU()(norm)

    return f

def BNA(_input):
    inputs_norm = BatchNormalization(mode=2, axis=1)(_input)
    return ELU()(inputs_norm)

def reduction_a(inputs, k=64, l=64, m=96, n=96):
    "35x35 -> 17x17"
    inputs_norm = BNA(inputs)
    pool1 = MaxPooling2D((3,3), strides=(2,2), border_mode='same')(inputs_norm)
    
    conv2 = Convolution2D(n, 3, 3, subsample=(2,2), border_mode='same')(inputs_norm)
    
    conv3_1 = NConvolution2D(k, 1, 1, subsample=(1,1), border_mode='same')(inputs_norm)
    conv3_2 = NConvolution2D(l, 3, 3, subsample=(1,1), border_mode='same')(conv3_1)
    conv3_2 = Convolution2D(m, 3, 3, subsample=(2,2), border_mode='same')(conv3_2)
    
    res = merge([pool1, conv2, conv3_2], mode='concat', concat_axis=1)
    return res


def reduction_b(inputs):
    "17x17 -> 8x8"
    inputs_norm = BNA(inputs)
    pool1 = MaxPooling2D((3,3), strides=(2,2), border_mode='same')(inputs_norm)
    #
    conv2_1 = NConvolution2D(64, 1, 1, subsample=(1,1), border_mode='same')(inputs_norm)
    conv2_2 = Convolution2D(96, 3, 3, subsample=(2,2), border_mode='same')(conv2_1)
    #
    conv3_1 = NConvolution2D(64, 1, 1, subsample=(1,1), border_mode='same')(inputs_norm)
    conv3_2 = Convolution2D(72, 3, 3, subsample=(2,2), border_mode='same')(conv3_1)
    #
    conv4_1 = NConvolution2D(64, 1, 1, subsample=(1,1), border_mode='same')(inputs_norm)
    conv4_2 = NConvolution2D(72, 3, 3, subsample=(1,1), border_mode='same')(conv4_1)
    conv4_3 = Convolution2D(80, 3, 3, subsample=(2,2), border_mode='same')(conv4_2)
    #
    res = merge([pool1, conv2_2, conv3_2, conv4_3], mode='concat', concat_axis=1)
    return res
    
    


def get_unet_inception_2head(optimizer):
    splitted = True
    act = 'elu'
    
    inputs = Input((1, IMG_ROWS, IMG_COLS), name='main_input')
    conv1 = inception_block(inputs, 32, batch_mode=2, splitted=splitted, activation=act)
    #conv1 = inception_block(conv1, 32, batch_mode=2, splitted=splitted, activation=act)
    
    #pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
    pool1 = NConvolution2D(32, 3, 3, border_mode='same', subsample=(2,2))(conv1)
    pool1 = Dropout(0.5)(pool1)
    
    conv2 = inception_block(pool1, 64, batch_mode=2, splitted=splitted, activation=act)
    #pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
    pool2 = NConvolution2D(64, 3, 3, border_mode='same', subsample=(2,2))(conv2)
    pool2 = Dropout(0.5)(pool2)
    
    conv3 = inception_block(pool2, 128, batch_mode=2, splitted=splitted, activation=act)
    #pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
    pool3 = NConvolution2D(128, 3, 3, border_mode='same', subsample=(2,2))(conv3)
    pool3 = Dropout(0.5)(pool3)
     
    conv4 = inception_block(pool3, 256, batch_mode=2, splitted=splitted, activation=act)
    #pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
    pool4 = NConvolution2D(256, 3, 3, border_mode='same', subsample=(2,2))(conv4)
    pool4 = Dropout(0.5)(pool4)
    
    conv5 = inception_block(pool4, 512, batch_mode=2, splitted=splitted, activation=act)
    #conv5 = inception_block(conv5, 512, batch_mode=2, splitted=splitted, activation=act)
    conv5 = Dropout(0.5)(conv5)
    
    #
    pre = Convolution2D(1, 1, 1, init='he_normal', activation='sigmoid')(conv5)
    pre = Flatten()(pre)
    aux_out = Dense(1, activation='sigmoid', name='aux_output')(pre) 
    #
    
    after_conv4 = rblock(conv4, 1, 256)
    up6 = merge([UpSampling2D(size=(2, 2))(conv5), after_conv4], mode='concat', concat_axis=1)
    conv6 = inception_block(up6, 256, batch_mode=2, splitted=splitted, activation=act)
    conv6 = Dropout(0.5)(conv6)
    
    after_conv3 = rblock(conv3, 1, 128)
    up7 = merge([UpSampling2D(size=(2, 2))(conv6), after_conv3], mode='concat', concat_axis=1)
    conv7 = inception_block(up7, 128, batch_mode=2, splitted=splitted, activation=act)
    conv7 = Dropout(0.5)(conv7)
    
    after_conv2 = rblock(conv2, 1, 64)
    up8 = merge([UpSampling2D(size=(2, 2))(conv7), after_conv2], mode='concat', concat_axis=1)
    conv8 = inception_block(up8, 64, batch_mode=2, splitted=splitted, activation=act)
    conv8 = Dropout(0.5)(conv8)
    
    after_conv1 = rblock(conv1, 1, 32)
    up9 = merge([UpSampling2D(size=(2, 2))(conv8), after_conv1], mode='concat', concat_axis=1)
    conv9 = inception_block(up9, 32, batch_mode=2, splitted=splitted, activation=act)
    #conv9 = inception_block(conv9, 32, batch_mode=2, splitted=splitted, activation=act)
    conv9 = Dropout(0.5)(conv9)

    conv10 = Convolution2D(1, 1, 1, init='he_normal', activation='sigmoid', name='main_output')(conv9)
    #print conv10._keras_shape

    model = Model(input=inputs, output=[conv10, aux_out])
    model.compile(optimizer=optimizer,
                  loss={'main_output': dice_coef_loss, 'aux_output': 'binary_crossentropy'},
                  metrics={'main_output': dice_coef, 'aux_output': 'acc'},
                  loss_weights={'main_output': 1., 'aux_output': 0.5})

    return model


get_unet = get_unet_inception_2head

def main():
    from keras.optimizers import Adam, RMSprop, SGD
    from metric import dice_coef, dice_coef_loss
    import numpy as np
    img_rows = IMG_ROWS
    img_cols = IMG_COLS
    
    optimizer = RMSprop(lr=0.045, rho=0.9, epsilon=1.0)
    model = get_unet(Adam(lr=1e-5))
    model.compile(optimizer=optimizer, loss=dice_coef_loss, metrics=[dice_coef])
    
    x = np.random.random((1, 1,img_rows,img_cols))
    res = model.predict(x, 1)
    print res
    #print 'res', res[0].shape
    print 'params', model.count_params()
    print 'layer num', len(model.layers)
    #


if __name__ == '__main__':
    sys.exit(main())



================================================
FILE: utils.py
================================================
import cPickle as pickle

def load_pickle(file_path):
    data = None
    with open (file_path,"rb") as dumpFile:
        data = pickle.load(dumpFile)
    return data

def save_pickle(file_path, data):
    with open (file_path,"wb") as dumpFile:
        pickle.dump(data, dumpFile, pickle.HIGHEST_PROTOCOL)

def count_enum(words):
    wdict = {}
    get = wdict.get
    for word in words:
        wdict[word] = get(word, 0) + 1
    return wdict