Full Code of deeplearningparis/dl-machine for AI

Repository: deeplearningparis/dl-machine
Branch: master
Commit: 3fb49592ac5b
Files: 11
Total size: 16.7 KB

Directory structure:
gitextract_zjr3qh4i/

├── LICENSE
├── README.md
├── caffe-Makefile.conf
├── circus.conf
├── circus.ini
├── numpy-site.cfg
├── scipy-site.cfg
├── scripts/
│   ├── install-deeplearning-libraries.sh
│   └── ubuntu-14.04-cuda-7.0.sh
├── theanorc
└── update-instance.conf

================================================
FILE CONTENTS
================================================

================================================
FILE: LICENSE
================================================
The MIT License (MIT)

Copyright (c) 2015 Olivier Grisel and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



================================================
FILE: README.md
================================================
# dl-machine

Scripts to setup a GPU / CUDA enable compute server with libraries to study deep learning development

## Setting up an Amazon g2.2xlarge spot instance

- log in to AWS management console and select EC2 instances
- select US-WEST (N. California) region in top left menu
- click on "Spot Request" on the leftmost menu and click "Request Spot Instances"
- select community AMIs and search for `ubuntu-14.04-hvm-deeplearning-paris`
- on the Choose instance Type tab, select GPU instances `g2.2xlarge`
- bid a price larger than current price (e.g. $0.30, if it fails check the spot pricing history for that instance type)
- in configure security group click Add Rule, and add a Custom TCP Rule with port Range `8888-8889` and from `Anywhere` 
- Review and launch, save the `mykey.pem` file
 
Once your machine is up (status : running in the online console), note the address to your instance :
 `INSTANCE_ID.compute.amazonaws.com`

Note: other regions with access to the deeplearning-paris image: Singapore, Ireland, North Virginia

## Start using your instance

#### Using the notebooks
By default an IPython notebook server and an iTorch notebook server should be running on port 8888 and 8889 respectively. You need to open those ports in the `Security Group` of your instance if you have not done so yet.

To start using your instance, simply open the following URLs in your favorite browser:

- http://INSTANCE_ID.compute.amazonaws.com:8888
- http://INSTANCE_ID.compute.amazonaws.com:8889


#### SSH Connection to your instance
Once the instance is up, you might need to access directly your instance via SSH:
- change the file `mykey.pem` accessibility:
```
chmod 400 mykey.pem
```
- ssh to your instance 
```
ssh -i mykey.pem ubuntu@INSTANCE_ID.compute.amazonaws.com
```

## Other instructions

#### Setting up ssh connection keys

This optional part helps you setting ssh connection keys for better and easier access to your instance. If you already have a public key, skip the keygen part.

- On your own generate a ssh key pair:
```
ssh-keygen
```
- Then go through the steps, you'll have two files, id_rsa and id_rsa.pub (the first is your private key, the second is your public key - the one you copy to remote machines)
```
cat ~/.ssh/id_rsa.pub
```
- On the remote instance, open authorized_keys file the and append your key id_rsa.pub 
```
vi ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
- On your local machine, you can then use a ssh-agent to store the decrypted key in memory so that you don't have to type the password each time. You can also add an alias in your `~/.ssh/config` file:
```
Host dlmachine
     HostName INSTANCE_ID.compute.amazonaws.com
     User ubuntu
     ServerAliveInterval 300
     ServerAliveCountMax 2
```
- You can now SSH to your machine with the following command:
```
ssh dlmachine
```

#### Running ipython / iTorch server

If the notebooks do not work you can login to your instance via as ssh:

```
ssh -A ubuntu@INSTANCE_ID.compute.amazonaws.com
```

(optional) Start a screen or tmux terminal:

```
screen
```
Use the `top` or `ps aux` command to check whether the ipython process is running. If this is not the case, launch the ipython and itorch notebook server:
```
ipython notebook --ip='*' --port=8888 --browser=none
itorch notebook --ip='*' --port=8889 --browser=none
```



================================================
FILE: caffe-Makefile.conf
================================================
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
		-gencode arch=compute_20,code=sm_21 \
		-gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
BLAS_INCLUDE := /opt/OpenBLAS-no-openmp/include
BLAS_LIB := /opt/OpenBLAS-no-openmp/bin

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
	       /home/ubuntu/venv/lib/python2.7/site-packages/numpy/core/include/numpy
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		# $(ANACONDA_HOME)/include/python2.7 \
		# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib/python2.7/config-x86_64-linux-gnu
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @


================================================
FILE: circus.conf
================================================
start on filesystem and net-device-up IFACE=lo
stop on runlevel [016]

respawn
exec /home/ubuntu/venv/bin/circusd /home/ubuntu/dl-machine/circus.ini


================================================
FILE: circus.ini
================================================
# This file is mean to be launched with the following command:
# circusd circus.ini
#
# It should start an ipython notebook server for the ubuntu user
# on port 8888
# You need to open that port in your EC2 Security Group
#
# Furthermore circusd offers a webconsole on port 8080 to monitor
# the resource usage of the managed processes.
[circus]
statsd = True
httpd = True
httpd_host = 0.0.0.0
httpd_port = 8080

[watcher:ipython-notebook]
uid = ubuntu
gid = ubuntu
working_dir = /home/ubuntu
cmd = /home/ubuntu/venv/bin/ipython
args = notebook --ip='*' --port=8888 --browser=none

# Log and rotate output
stdout_stream.class = FileStream
stdout_stream.filename = /home/ubuntu/ipython-out.log
stdout_stream.max_bytes = 1073741824
stdout_stream.backup_count = 5
stderr_stream.class = FileStream
stderr_stream.filename = /home/ubuntu/ipython-err.log
stderr_stream.max_bytes = 1073741824
stderr_stream.backup_count = 5

[env:ipython-notebook]
PATH=/home/ubuntu/venv/bin:/usr/local/cuda/bin:$PATH
LD_LIBRARY_PATH=/usr/local/cuda:/usr/local/cuda/lib64:/opt/OpenBLAS-no-openmp/lib


[watcher:itorch-notebook]
uid = ubuntu
gid = ubuntu
working_dir = /home/ubuntu
cmd = /home/ubuntu/torch/install/bin/itorch
args = notebook --ip='*' --port=8889 --browser=none

# Log and rotate output
stdout_stream.class = FileStream
stdout_stream.filename = /home/ubuntu/itorch-out.log
stdout_stream.max_bytes = 1073741824
stdout_stream.backup_count = 5
stderr_stream.class = FileStream
stderr_stream.filename = /home/ubuntu/itorch-err.log
stderr_stream.max_bytes = 1073741824
stderr_stream.backup_count = 5

[env:itorch-notebook]
PATH=/home/ubuntu/torch/bin:/home/ubuntu/venv/bin:/usr/local/cuda/bin:$PATH
LD_LIBRARY_PATH=/usr/local/cuda:/usr/local/cuda/lib64:/opt/OpenBLAS/lib:/home/ubuntu/torch/lib



================================================
FILE: numpy-site.cfg
================================================
[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS-no-openmp/lib
include_dirs = /opt/OpenBLAS-no-openmp/include



================================================
FILE: scipy-site.cfg
================================================
[DEFAULT]
library_dirs = /opt/OpenBLAS-no-openmp/lib:/usr/local/lib
include_dirs = /opt/OpenBLAS-no-openmp/include:/usr/local/include

[blas_opt]
libraries = openblas

[lapack_opt]
libraries = openblas



================================================
FILE: scripts/install-deeplearning-libraries.sh
================================================
#!/usr/bin/env bash
# This script will try to install recent versions of common open source
# tools for deep learning. This requires a Ubuntu 14.04 instance with
# a recent version of CUDA. See "ubuntu-14.04-cuda-6.5.sh" to build setup
# an AWS EC2 g2.2xlarge instance type for instance.
set -xe
cd $HOME


# Check that the NVIDIA drivers are installed properly and the GPU is in a
# good shape:
nvidia-smi

# Build latest stable release of OpenBLAS without OPENMP to make it possible
# to use Python multiprocessing and forks without crash
# The torch install script will install OpenBLAS with OPENMP enabled in
# /opt/OpenBLAS so we need to install the OpenBLAS used by Python in a
# distinct folder.
# Note: the master branch only has the release tags in it
sudo apt-get install -y gfortran
export OPENBLAS_ROOT=/opt/OpenBLAS-no-openmp
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OPENBLAS_ROOT/lib
if [ ! -d "OpenBLAS" ]; then
    git clone -q --branch=master git://github.com/xianyi/OpenBLAS.git
    (cd OpenBLAS \
      && make FC=gfortran USE_OPENMP=0 NO_AFFINITY=1 NUM_THREADS=32 \
      && sudo make install PREFIX=$OPENBLAS_ROOT)
    echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> ~/.bashrc
fi
sudo ldconfig

# Python basics: update pip and setup a virtualenv to avoid mixing packages
# installed from source with system packages
sudo apt-get update
sudo apt-get install -y python-dev python-pip htop
sudo pip install -U pip virtualenv
if [ ! -d "venv" ]; then
    virtualenv venv
    echo "source ~/venv/bin/activate" >> ~/.bashrc
fi
source venv/bin/activate
pip install -U pip
pip install -U circus circus-web Cython Pillow

# Checkout this project to access installation script and additional resources
if [ ! -d "dl-machine" ]; then
    git clone https://github.com:deeplearningparis/dl-machine.git
else
    if  [ "$1" == "reset" ]; then
        (cd dl-machine && git reset --hard && git checkout master && git pull --rebase origin master)
    fi
fi

# Build numpy from source against OpenBLAS
# You might need to install liblapack-dev package as well
# sudo apt-get install -y liblapack-dev
rm -f ~/.numpy-site.cfg
ln -s dl-machine/numpy-site.cfg ~/.numpy-site.cfg
pip install -U numpy

# Build scipy from source against OpenBLAS
rm -f ~/.scipy-site.cfg
ln -s dl-machine/scipy-site.cfg ~/.scipy-site.cfg
pip install -U scipy

# Install common tools from the scipy stack
sudo apt-get install -y libfreetype6-dev libpng12-dev
pip install -U matplotlib ipython[all] pandas scikit-image

# Scikit-learn (generic machine learning utilities)
pip install -e git+git://github.com/scikit-learn/scikit-learn.git#egg=scikit-learn

# Theano
pip install -e git+git://github.com/Theano/Theano.git#egg=Theano
if [ ! -f ".theanorc" ]; then
    ln -s dl-machine/theanorc ~/.theanorc
fi

# Tutorial files
if [ ! -d "DL4H" ]; then
    git clone https://github.com/SnippyHolloW/DL4H.git
else
    if  [ "$1" == "reset" ]; then
        (cd DL4H && git reset --hard && git checkout master && git pull --rebase origin master)
    fi
fi

# Keras (will be using theano by default)
if [ ! -d "keras" ]; then
    git clone https://github.com/fchollet/keras.git
    (cd keras && python setup.py install)
else
    if  [ "$1" == "reset" ]; then
	(cd keras && git reset --hard && git checkout master && git pull --rebase $REMOTE master && python setup.py install)
    fi
fi

# Tensorflow (cpu mode only, GPU not officially supported on AWS - CUDA 3.0 architecture)
pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl

# Torch
if [ ! -d "torch" ]; then
    curl -sk https://raw.githubusercontent.com/torch/ezinstall/master/install-deps | bash
    git clone https://github.com/torch/distro.git ~/torch --recursive
    (cd ~/torch && yes | ./install.sh)
fi
. ~/torch/install/bin/torch-activate

if [ ! -d "iTorch" ]; then
    git clone https://github.com/facebook/iTorch.git
else
    if  [ "$1" == "reset" ]; then
        (cd iTorch && git reset --hard && git checkout master && git pull --rebase origin master)
    fi
fi
(cd iTorch && luarocks make)


# Install caffe

sudo apt-get install -y protobuf-compiler libboost-all-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libsnappy-dev libopencv-dev libyaml-dev libprotobuf-dev

if [ ! -d "caffe" ]; then
    git clone https://github.com/BVLC/caffe.git
    (cd caffe && cp $HOME/dl-machine/caffe-Makefile.conf Makefile.conf && cmake -DBLAS=open . && make all)
    (cd caffe/python && pip install -r requirements.txt)
else
    if [ "$1" == "reset" ]; then
	(cd caffe && git reset --hard && git checkout master && git pull --rebase origin master && cp $HOME/dl-machine/caffe-Makefile.conf Makefile.conf && cmake -DBLAS=open . && make all)
    fi
fi


# Register the circus daemon with Upstart
if [ ! -f "/etc/init/circus.conf" ]; then
    sudo ln -s $HOME/dl-machine/circus.conf /etc/init/circus.conf
    sudo initctl reload-configuration
fi
sudo service circus restart


# Register a task job to get the main repo of the image automatically up to date
# at boot time
if [ ! -f "/etc/init/update-instance.conf" ]; then
    sudo ln -s $HOME/dl-machine/update-instance.conf /etc/init/update-instance.conf
fi


================================================
FILE: scripts/ubuntu-14.04-cuda-7.0.sh
================================================
#!/usr/bin/env bash
# Script to build an Ubuntu-based g2.2xlarge with CUDA 7.0 enabled.

export DEBIAN_FRONTEND=noninteractive
sudo apt-get update -y
sudo apt-get -y dist-upgrade
sudo apt-get install -y git wget linux-image-generic build-essential

cd /tmp
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.0-28_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_7.0-28_amd64.deb
sudo apt-get update -y
sudo apt-get install -y cuda

echo -e "\nexport CUDA_HOME=/usr/local/cuda\nexport CUDA_ROOT=/usr/local/cuda" >> ~/.bashrc
echo -e "\nexport PATH=/usr/local/cuda/bin:\$PATH\nexport LD_LIBRARY_PATH=/usr/local/cuda/lib64:\$LD_LIBRARY_PATH" >> ~/.bashrc

echo "CUDA installation complete: rebooting the instance now!"
sudo reboot


================================================
FILE: theanorc
================================================
[global]
device=gpu
floatX=float32
allow_gc=False
optimizer_including="local_ultra_fast_sigmoid"

[nvcc]
use_fast_math=True
fastmath=True


================================================
FILE: update-instance.conf
================================================
start on runlevel [2345]
stop on runlevel [!2345]

task

script
  su - ubuntu -c 'bash /home/ubuntu/dl-machine/scripts/install-deeplearning-libraries.sh reset'
end script