[
  {
    "path": ".gitignore",
    "content": ".idea\n__pycache__\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2019 Robotics and Perception Group\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# Event Representation Learning\n\n[![Event Representation Learning](resources/youtube_preview.png)](https://youtu.be/bQtSx59GXRY)\n\nThis repository contains learning code that implements \nevent representation learning as described in Gehrig et al. ICCV'19. The paper can be found [here](http://rpg.ifi.uzh.ch/docs/ICCV19_Gehrig.pdf)\n\nIf you use this code in an academic context, please cite the following work:\n\n[Daniel Gehrig](https://danielgehrig18.github.io/), [Antonio Loquercio](https://antonilo.github.io/), Konstantinos G. Derpanis, Davide Scaramuzza, \"End-to-End Learning of Representations \nfor Asynchronous Event-Based Data\", The International Conference on Computer Vision (ICCV), 2019\n\n```bibtex\n@InProceedings{Gehrig_2019_ICCV,\n  author = {Daniel Gehrig and Antonio Loquercio and Konstantinos G. Derpanis and Davide Scaramuzza},\n  title = {End-to-End Learning of Representations for Asynchronous Event-Based Data},\n  booktitle = {Int. Conf. Comput. Vis. (ICCV)},\n  month = {October},\n  year = {2019}\n}\n```\n\n## Requirements\n\n* Python 3.7\n* virtualenv\n* cuda 10\n\n## Dependencies\nCreate a virtual environment with `python3.7` and activate it\n\n    virtualenv venv -p /usr/local/bin/python3.7\n    source venv/bin/activate\n\nInstall all dependencies by calling \n\n    pip install -r requirements.txt\n   \n## Training\nBefore training, download the `N-Caltech101` dataset and unzip it\n\n    wget http://rpg.ifi.uzh.ch/datasets/gehrig_et_al_iccv19/N-Caltech101.zip \n    unzip N-Caltech101.zip\n    \nThen start training by calling\n\n    python main.py --validation_dataset N-Caltech101/validation/ --training_dataset N-Caltech101/training/ --log_dir log/temp --device cuda:0\n\nHere, `validation_dataset` and `training_dataset` should point to the folders where the training and validation set are stored.\n`log_dir` controls logging and `device` controls on which device you want to train. Checkpoints and models with lowest validation loss will be saved in the root folder of `log_dir`.\n\nThe N-Cars dataset can be downloaded [here](http://rpg.ifi.uzh.ch/datasets/gehrig_et_al_iccv19/N-Cars.zip).\n\n### Additional parameters \n* `--num_worker` how many threads to use to load data\n* `--pin_memory` wether to pin memory or not\n* `--num_epochs` number of epochs to train\n* `--save_every_n_epochs` save a checkpoint every n epochs.\n* `--batch_size` batch size for training\n\n### Visualization\n\nTraining can be visualized by calling tensorboard\n\n    tensorboard --logdir log/temp\n\nTraining and validation losses as well as classification accuracies are plotted. In addition, the learnt representations are visualized. The training and validation curves should look something like this:    \n![alt_text](resources/tb.png)\n\n## Testing\nOnce trained, the models can be tested by calling the following script:\n\n    python testing.py --test N-Caltech101/testing/ --device cuda:0\n\nWhich will print the test score after iteration through the whole dataset.\n\n    \n"
  },
  {
    "path": "log/README.md",
    "content": "logs will be written here."
  },
  {
    "path": "main.py",
    "content": "import argparse\nfrom os.path import dirname\nimport torch\nimport torchvision\nimport os\nimport numpy as np\nimport tqdm\n\nfrom utils.models import Classifier\nfrom torch.utils.tensorboard import SummaryWriter\nfrom utils.loader import Loader\nfrom utils.loss import cross_entropy_loss_and_accuracy\nfrom utils.dataset import NCaltech101\n\n\ntorch.manual_seed(1)\nnp.random.seed(1)\n\n\ndef FLAGS():\n    parser = argparse.ArgumentParser(\"\"\"Train classifier using a learnt quantization layer.\"\"\")\n\n    # training / validation dataset\n    parser.add_argument(\"--validation_dataset\", default=\"\", required=True)\n    parser.add_argument(\"--training_dataset\", default=\"\", required=True)\n\n    # logging options\n    parser.add_argument(\"--log_dir\", default=\"\", required=True)\n\n    # loader and device options\n    parser.add_argument(\"--device\", default=\"cuda:0\")\n    parser.add_argument(\"--num_workers\", type=int, default=4)\n    parser.add_argument(\"--pin_memory\", type=bool, default=True)\n    parser.add_argument(\"--batch_size\", type=int, default=4)\n\n    parser.add_argument(\"--num_epochs\", type=int, default=30)\n    parser.add_argument(\"--save_every_n_epochs\", type=int, default=5)\n\n    flags = parser.parse_args()\n\n    assert os.path.isdir(dirname(flags.log_dir)), f\"Log directory root {dirname(flags.log_dir)} not found.\"\n    assert os.path.isdir(flags.validation_dataset), f\"Validation dataset directory {flags.validation_dataset} not found.\"\n    assert os.path.isdir(flags.training_dataset), f\"Training dataset directory {flags.training_dataset} not found.\"\n\n    print(f\"----------------------------\\n\"\n          f\"Starting training with \\n\"\n          f\"num_epochs: {flags.num_epochs}\\n\"\n          f\"batch_size: {flags.batch_size}\\n\"\n          f\"device: {flags.device}\\n\"\n          f\"log_dir: {flags.log_dir}\\n\"\n          f\"training_dataset: {flags.training_dataset}\\n\"\n          f\"validation_dataset: {flags.validation_dataset}\\n\"\n          f\"----------------------------\")\n\n    return flags\n\ndef percentile(t, q):\n    B, C, H, W = t.shape\n    k = 1 + round(.01 * float(q) * (C * H * W - 1))\n    result = t.view(B, -1).kthvalue(k).values\n    return result[:,None,None,None]\n\ndef create_image(representation):\n    B, C, H, W = representation.shape\n    representation = representation.view(B, 3, C // 3, H, W).sum(2)\n\n    # do robust min max norm\n    representation = representation.detach().cpu()\n    robust_max_vals = percentile(representation, 99)\n    robust_min_vals = percentile(representation, 1)\n\n    representation = (representation - robust_min_vals)/(robust_max_vals - robust_min_vals)\n    representation = torch.clamp(255*representation, 0, 255).byte()\n\n    representation = torchvision.utils.make_grid(representation)\n\n    return representation\n\n\nif __name__ == '__main__':\n    flags = FLAGS()\n\n    # datasets, add augmentation to training set\n    training_dataset = NCaltech101(flags.training_dataset, augmentation=True)\n    validation_dataset = NCaltech101(flags.validation_dataset)\n\n    # construct loader, handles data streaming to gpu\n    training_loader = Loader(training_dataset, flags, device=flags.device)\n    validation_loader = Loader(validation_dataset, flags, device=flags.device)\n\n    # model, and put to device\n    model = Classifier()\n    model = model.to(flags.device)\n\n    # optimizer and lr scheduler\n    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)\n    lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, 0.5)\n\n    writer = SummaryWriter(flags.log_dir)\n\n    iteration = 0\n    min_validation_loss = 1000\n\n    for i in range(flags.num_epochs):\n        sum_accuracy = 0\n        sum_loss = 0\n        model = model.eval()\n\n        print(f\"Validation step [{i:3d}/{flags.num_epochs:3d}]\")\n        for events, labels in tqdm.tqdm(validation_loader):\n\n            with torch.no_grad():\n                pred_labels, representation = model(events)\n                loss, accuracy = cross_entropy_loss_and_accuracy(pred_labels, labels)\n\n            sum_accuracy += accuracy\n            sum_loss += loss\n\n        validation_loss = sum_loss.item() / len(validation_loader)\n        validation_accuracy = sum_accuracy.item() / len(validation_loader)\n\n        writer.add_scalar(\"validation/accuracy\", validation_accuracy, iteration)\n        writer.add_scalar(\"validation/loss\", validation_loss, iteration)\n\n        # visualize representation\n        representation_vizualization = create_image(representation)\n        writer.add_image(\"validation/representation\", representation_vizualization, iteration)\n\n        print(f\"Validation Loss {validation_loss:.4f}  Accuracy {validation_accuracy:.4f}\")\n\n        if validation_loss < min_validation_loss:\n            min_validation_loss = validation_loss\n            state_dict = model.state_dict()\n\n            torch.save({\n                \"state_dict\": state_dict,\n                \"min_val_loss\": min_validation_loss,\n                \"iteration\": iteration\n            }, \"log/model_best.pth\")\n            print(\"New best at \", validation_loss)\n\n        if i % flags.save_every_n_epochs == 0:\n            state_dict = model.state_dict()\n            torch.save({\n                \"state_dict\": state_dict,\n                \"min_val_loss\": min_validation_loss,\n                \"iteration\": iteration\n            }, \"log/checkpoint_%05d_%.4f.pth\" % (iteration, min_validation_loss))\n\n        sum_accuracy = 0\n        sum_loss = 0\n\n        model = model.train()\n        print(f\"Training step [{i:3d}/{flags.num_epochs:3d}]\")\n        for events, labels in tqdm.tqdm(training_loader):\n            optimizer.zero_grad()\n\n            pred_labels, representation = model(events)\n            loss, accuracy = cross_entropy_loss_and_accuracy(pred_labels, labels)\n\n            loss.backward()\n\n            optimizer.step()\n\n            sum_accuracy += accuracy\n            sum_loss += loss\n\n            iteration += 1\n\n        if i % 10 == 9:\n            lr_scheduler.step()\n\n        training_loss = sum_loss.item() / len(training_loader)\n        training_accuracy = sum_accuracy.item() / len(training_loader)\n        print(f\"Training Iteration {iteration:5d}  Loss {training_loss:.4f}  Accuracy {training_accuracy:.4f}\")\n\n        writer.add_scalar(\"training/accuracy\", training_accuracy, iteration)\n        writer.add_scalar(\"training/loss\", training_loss, iteration)\n\n        representation_vizualization = create_image(representation)\n        writer.add_image(\"training/representation\", representation_vizualization, iteration)\n"
  },
  {
    "path": "requirements.txt",
    "content": "https://download.pytorch.org/whl/cu100/torch-1.1.0-cp37-cp37m-linux_x86_64.whl\nhttps://download.pytorch.org/whl/cu100/torchvision-0.3.0-cp37-cp37m-linux_x86_64.whl\ntqdm\ntb-nightly\nfuture"
  },
  {
    "path": "testing.py",
    "content": "from os.path import dirname\nimport argparse\nimport torch\nimport tqdm\nimport os\n\nfrom loader import Loader\nfrom loss import cross_entropy_loss_and_accuracy\nfrom models import Classifier\nfrom dataset import NCaltech101\n\n\ndef FLAGS():\n    parser = argparse.ArgumentParser(\n        \"\"\"Deep Learning for Events. Supply a config file.\"\"\")\n\n    # can be set in config\n    parser.add_argument(\"--checkpoint\", default=\"\", required=True)\n    parser.add_argument(\"--test_dataset\", default=\"\", required=True)\n    parser.add_argument(\"--device\", default=\"cuda:0\")\n    parser.add_argument(\"--num_workers\", type=int, default=4)\n    parser.add_argument(\"--batch_size\", type=int, default=4)\n    parser.add_argument(\"--pin_memory\", type=bool, default=True)\n\n    flags = parser.parse_args()\n\n    assert os.path.isdir(dirname(flags.checkpoint)), f\"Checkpoint{flags.checkpoint} not found.\"\n    assert os.path.isdir(flags.test_dataset), f\"Test dataset directory {flags.test_dataset} not found.\"\n\n    print(f\"----------------------------\\n\"\n          f\"Starting testing with \\n\"\n          f\"checkpoint: {flags.checkpoint}\\n\"\n          f\"test_dataset: {flags.test_dataset}\\n\"\n          f\"batch_size: {flags.batch_size}\\n\"\n          f\"device: {flags.device}\\n\"\n          f\"----------------------------\")\n\n    return flags\n\n\nif __name__ == '__main__':\n    flags = FLAGS()\n\n    test_dataset = NCaltech101(flags.test_dataset)\n\n    # construct loader, responsible for streaming data to gpu\n    test_loader = Loader(test_dataset, flags, flags.device)\n\n    # model, load and put to device\n    model = Classifier()\n    ckpt = torch.load(flags.checkpoint)\n    model.load_state_dict(ckpt[\"state_dict\"])\n    model = model.to(flags.device)\n\n    model = model.eval()\n    sum_accuracy = 0\n    sum_loss = 0\n\n    print(\"Test step\")\n    for events, labels in tqdm.tqdm(test_loader):\n        with torch.no_grad():\n            pred_labels, _ = model(events)\n            loss, accuracy = cross_entropy_loss_and_accuracy(pred_labels, labels)\n\n        sum_accuracy += accuracy\n        sum_loss += loss\n\n    test_loss = sum_loss.item() / len(test_loader)\n    test_accuracy = sum_accuracy.item() / len(test_loader)\n\n    print(f\"Test Loss: {test_loss}, Test Accuracy: {test_accuracy}\")"
  },
  {
    "path": "utils/dataset.py",
    "content": "import numpy as np\nfrom os import listdir\nfrom os.path import join\n\n\ndef random_shift_events(events, max_shift=20, resolution=(180, 240)):\n    H, W = resolution\n    x_shift, y_shift = np.random.randint(-max_shift, max_shift+1, size=(2,))\n    events[:,0] += x_shift\n    events[:,1] += y_shift\n\n    valid_events = (events[:,0] >= 0) & (events[:,0] < W) & (events[:,1] >= 0) & (events[:,1] < H)\n    events = events[valid_events]\n\n    return events\n\ndef random_flip_events_along_x(events, resolution=(180, 240), p=0.5):\n    H, W = resolution\n    if np.random.random() < p:\n        events[:,0] = W - 1 - events[:,0]\n    return events\n\n\nclass NCaltech101:\n    def __init__(self, root, augmentation=False):\n        self.classes = listdir(root)\n\n        self.files = []\n        self.labels = []\n\n        self.augmentation = augmentation\n\n        for i, c in enumerate(self.classes):\n            new_files = [join(root, c, f) for f in listdir(join(root, c))]\n            self.files += new_files\n            self.labels += [i] * len(new_files)\n\n    def __len__(self):\n        return len(self.files)\n\n    def __getitem__(self, idx):\n        \"\"\"\n        returns events and label, loading events from aedat\n        :param idx:\n        :return: x,y,t,p,  label\n        \"\"\"\n        label = self.labels[idx]\n        f = self.files[idx]\n        events = np.load(f).astype(np.float32)\n\n        if self.augmentation:\n            events = random_shift_events(events)\n            events = random_flip_events_along_x(events)\n\n        return events, label"
  },
  {
    "path": "utils/loader.py",
    "content": "import torch\nimport numpy as np\n\nfrom torch.utils.data.dataloader import default_collate\n\n\nclass Loader:\n    def __init__(self, dataset, flags, device):\n        self.device = device\n        split_indices = list(range(len(dataset)))\n        sampler = torch.utils.data.sampler.SubsetRandomSampler(split_indices)\n        self.loader = torch.utils.data.DataLoader(dataset, batch_size=flags.batch_size, sampler=sampler,\n                                             num_workers=flags.num_workers, pin_memory=flags.pin_memory,\n                                             collate_fn=collate_events)\n\n    def __iter__(self):\n        for data in self.loader:\n            data = [d.to(self.device) for d in data]\n            yield data\n\n    def __len__(self):\n        return len(self.loader)\n\n\ndef collate_events(data):\n    labels = []\n    events = []\n    for i, d in enumerate(data):\n        labels.append(d[1])\n        ev = np.concatenate([d[0], i*np.ones((len(d[0]),1), dtype=np.float32)],1)\n        events.append(ev)\n    events = torch.from_numpy(np.concatenate(events,0))\n    labels = default_collate(labels)\n    return events, labels"
  },
  {
    "path": "utils/loss.py",
    "content": "import torch\n\n\ndef cross_entropy_loss_and_accuracy(prediction, target):\n    cross_entropy_loss = torch.nn.CrossEntropyLoss()\n    loss = cross_entropy_loss(prediction, target)\n    accuracy = (prediction.argmax(1) == target).float().mean()\n    return loss, accuracy"
  },
  {
    "path": "utils/models.py",
    "content": "import torch.nn as nn\nfrom os.path import join, dirname, isfile\nimport torch\nimport torch.nn.functional as F\nimport numpy as np\nfrom torchvision.models.resnet import resnet34\nimport tqdm\n\n\nclass ValueLayer(nn.Module):\n    def __init__(self, mlp_layers, activation=nn.ReLU(), num_channels=9):\n        assert mlp_layers[-1] == 1, \"Last layer of the mlp must have 1 input channel.\"\n        assert mlp_layers[0] == 1, \"First layer of the mlp must have 1 output channel\"\n\n        nn.Module.__init__(self)\n        self.mlp = nn.ModuleList()\n        self.activation = activation\n\n        # create mlp\n        in_channels = 1\n        for out_channels in mlp_layers[1:]:\n            self.mlp.append(nn.Linear(in_channels, out_channels))\n            in_channels = out_channels\n\n        # init with trilinear kernel\n        path = join(dirname(__file__), \"quantization_layer_init\", \"trilinear_init.pth\")\n        if isfile(path):\n            state_dict = torch.load(path)\n            self.load_state_dict(state_dict)\n        else:\n            self.init_kernel(num_channels)\n\n    def forward(self, x):\n        # create sample of batchsize 1 and input channels 1\n        x = x[None,...,None]\n\n        # apply mlp convolution\n        for i in range(len(self.mlp[:-1])):\n            x = self.activation(self.mlp[i](x))\n\n        x = self.mlp[-1](x)\n        x = x.squeeze()\n\n        return x\n\n    def init_kernel(self, num_channels):\n        ts = torch.zeros((1, 2000))\n        optim = torch.optim.Adam(self.parameters(), lr=1e-2)\n\n        torch.manual_seed(1)\n\n        for _ in tqdm.tqdm(range(1000)):  # converges in a reasonable time\n            optim.zero_grad()\n\n            ts.uniform_(-1, 1)\n\n            # gt\n            gt_values = self.trilinear_kernel(ts, num_channels)\n\n            # pred\n            values = self.forward(ts)\n\n            # optimize\n            loss = (values - gt_values).pow(2).sum()\n\n            loss.backward()\n            optim.step()\n\n\n    def trilinear_kernel(self, ts, num_channels):\n        gt_values = torch.zeros_like(ts)\n\n        gt_values[ts > 0] = (1 - (num_channels-1) * ts)[ts > 0]\n        gt_values[ts < 0] = ((num_channels-1) * ts + 1)[ts < 0]\n\n        gt_values[ts < -1.0 / (num_channels-1)] = 0\n        gt_values[ts > 1.0 / (num_channels-1)] = 0\n\n        return gt_values\n\n\nclass QuantizationLayer(nn.Module):\n    def __init__(self, dim,\n                 mlp_layers=[1, 100, 100, 1],\n                 activation=nn.LeakyReLU(negative_slope=0.1)):\n        nn.Module.__init__(self)\n        self.value_layer = ValueLayer(mlp_layers,\n                                      activation=activation,\n                                      num_channels=dim[0])\n        self.dim = dim\n\n    def forward(self, events):\n        # points is a list, since events can have any size\n        B = int((1+events[-1,-1]).item())\n        num_voxels = int(2 * np.prod(self.dim) * B)\n        vox = events[0].new_full([num_voxels,], fill_value=0)\n        C, H, W = self.dim\n\n        # get values for each channel\n        x, y, t, p, b = events.t()\n\n        # normalizing timestamps\n        for bi in range(B):\n            t[events[:,-1] == bi] /= t[events[:,-1] == bi].max()\n\n        p = (p+1)/2  # maps polarity to 0, 1\n\n        idx_before_bins = x \\\n                          + W * y \\\n                          + 0 \\\n                          + W * H * C * p \\\n                          + W * H * C * 2 * b\n\n        for i_bin in range(C):\n            values = t * self.value_layer.forward(t-i_bin/(C-1))\n\n            # draw in voxel grid\n            idx = idx_before_bins + W * H * i_bin\n            vox.put_(idx.long(), values, accumulate=True)\n\n        vox = vox.view(-1, 2, C, H, W)\n        vox = torch.cat([vox[:, 0, ...], vox[:, 1, ...]], 1)\n\n        return vox\n\n\nclass Classifier(nn.Module):\n    def __init__(self,\n                 voxel_dimension=(9,180,240),  # dimension of voxel will be C x 2 x H x W\n                 crop_dimension=(224, 224),  # dimension of crop before it goes into classifier\n                 num_classes=101,\n                 mlp_layers=[1, 30, 30, 1],\n                 activation=nn.LeakyReLU(negative_slope=0.1),\n                 pretrained=True):\n\n        nn.Module.__init__(self)\n        self.quantization_layer = QuantizationLayer(voxel_dimension, mlp_layers, activation)\n        self.classifier = resnet34(pretrained=pretrained)\n\n        self.crop_dimension = crop_dimension\n\n        # replace fc layer and first convolutional layer\n        input_channels = 2*voxel_dimension[0]\n        self.classifier.conv1 = nn.Conv2d(input_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)\n        self.classifier.fc = nn.Linear(self.classifier.fc.in_features, num_classes)\n\n    def crop_and_resize_to_resolution(self, x, output_resolution=(224, 224)):\n        B, C, H, W = x.shape\n        if H > W:\n            h = H // 2\n            x = x[:, :, h - W // 2:h + W // 2, :]\n        else:\n            h = W // 2\n            x = x[:, :, :, h - H // 2:h + H // 2]\n\n        x = F.interpolate(x, size=output_resolution)\n\n        return x\n\n    def forward(self, x):\n        vox = self.quantization_layer.forward(x)\n        vox_cropped = self.crop_and_resize_to_resolution(vox, self.crop_dimension)\n        pred = self.classifier.forward(vox_cropped)\n        return pred, vox\n\n\n"
  }
]