/cnn; python test.py --model_path bananas.pt
```
The error on the test set should be 2.57%. This can be run on a CPU or GPU, but it will be faster on a GPU.
The best neural architecture found by BANANAS on CIFAR-10. Convolutional cell (left), and reduction cell (right).
## Train BANANAS architecture
Train the best architecture found by BANANAS.
```bash
cd /cnn; python train.py --auxiliary --cutout
```
This will train the architecture from scratch, which takes about 34 hours on an NVIDIA V100 GPU.
The final test error should be 2.59%.
Setting the random seed to 4 by adding `--seed 4` will result in a test error of 2.57%.
We report the random seeds and hardware used in Table 2 of our paper [here](https://docs.google.com/spreadsheets/d/1z6bHUgX8r0y9Bh9Zxot_B9nT_9qLWJoD0Um0fTYdpus/edit?usp=sharing).
## Run BANANAS on the NASBench search space
To run BANANAS on NASBench, download `nasbench_only108.tfrecord` and place it in the top level folder of this repo.
```bash
python run_experiments_sequential.py
```
This will test the nasbench algorithm against several other NAS algorithms on the NASBench search space.
To customize your experiment, open `params.py`. Here, you can change the hyperparameters and the algorithms to run.
To run experiments with NAS-Bench-201, download `NAS-Bench-201-v1_0-e61699.pth` and place it in the top level folder of this repo.
Choose between cifar10, cifar100, and imagenet. For example,
```bash
python run_experiments_sequential.py --search_space nasbench_201_cifar10
```
## Run BANANAS on the DARTS search space
We highly recommend using multiple GPUs to run BANANAS on the DARTS search space. You can run BANANAS in parallel on GCP using the shell script:
```bash
run_experiments_parallel.sh
```
## Contributions
We welcome community contributions to this repo!
## Citation
Please cite [our paper](https://arxiv.org/abs/1910.11858) if you use code from this repo:
```bibtex
@inproceedings{white2019bananas,
title={BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search},
author={White, Colin and Neiswanger, Willie and Savani, Yash},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2021}
}
```
================================================
FILE: acquisition_functions.py
================================================
import numpy as np
import sys
# Different acquisition functions that can be used by BANANAS
def acq_fn(predictions, explore_type='its'):
predictions = np.array(predictions)
# Upper confidence bound (UCB) acquisition function
if explore_type == 'ucb':
explore_factor = 0.5
mean = np.mean(predictions, axis=0)
std = np.sqrt(np.var(predictions, axis=0))
ucb = mean - explore_factor * std
sorted_indices = np.argsort(ucb)
# Expected improvement (EI) acquisition function
elif explore_type == 'ei':
ei_calibration_factor = 5.
mean = list(np.mean(predictions, axis=0))
std = list(np.sqrt(np.var(predictions, axis=0)) /
ei_calibration_factor)
min_y = ytrain.min()
gam = [(min_y - mean[i]) / std[i] for i in range(len(mean))]
ei = [-1 * std[i] * (gam[i] * norm.cdf(gam[i]) + norm.pdf(gam[i]))
for i in range(len(mean))]
sorted_indices = np.argsort(ei)
# Probability of improvement (PI) acquisition function
elif explore_type == 'pi':
mean = list(np.mean(predictions, axis=0))
std = list(np.sqrt(np.var(predictions, axis=0)))
min_y = ytrain.min()
pi = [-1 * norm.cdf(min_y, loc=mean[i], scale=std[i]) for i in range(len(mean))]
sorted_indices = np.argsort(pi)
# Thompson sampling (TS) acquisition function
elif explore_type == 'ts':
rand_ind = np.random.randint(predictions.shape[0])
ts = predictions[rand_ind,:]
sorted_indices = np.argsort(ts)
# Top exploitation
elif explore_type == 'percentile':
min_prediction = np.min(predictions, axis=0)
sorted_indices = np.argsort(min_prediction)
# Top mean
elif explore_type == 'mean':
mean = np.mean(predictions, axis=0)
sorted_indices = np.argsort(mean)
elif explore_type == 'confidence':
confidence_factor = 2
mean = np.mean(predictions, axis=0)
std = np.sqrt(np.var(predictions, axis=0))
conf = mean + confidence_factor * std
sorted_indices = np.argsort(conf)
# Independent Thompson sampling (ITS) acquisition function
elif explore_type == 'its':
mean = np.mean(predictions, axis=0)
std = np.sqrt(np.var(predictions, axis=0))
samples = np.random.normal(mean, std)
sorted_indices = np.argsort(samples)
else:
print('Invalid exploration type in meta neuralnet search', explore_type)
sys.exit()
return sorted_indices
================================================
FILE: bo/__init__.py
================================================
"""
Code for running Bayesian Optimization (BO) in NASzilla.
"""
================================================
FILE: bo/acq/__init__.py
================================================
"""
Code for acquisition strategies.
"""
================================================
FILE: bo/acq/acqmap.py
================================================
"""
Classes to manage acqmap (acquisition maps from xin to acquisition value).
"""
from argparse import Namespace
import numpy as np
import copy
from bo.acq.acquisition import Acquisitioner
from bo.util.datatransform import DataTransformer
#from bo.pp.pp_gp_george import GeorgeGpPP
#from bo.pp.pp_gp_stan import StanGpPP
from bo.pp.pp_gp_my_distmat import MyGpDistmatPP
class AcqMapper(object):
""" Class to manage acqmap (acquisition map). """
def __init__(self, data, amp, print_flag=True):
""" Constructor
Parameters:
amp - Namespace of acqmap params
print_flag - True or False
"""
self.data = data
self.set_am_params(amp)
#self.setup_acqmap()
if print_flag: self.print_str()
def set_am_params(self, amp):
""" Set the acqmap params.
Inputs:
amp - Namespace of acqmap parameters """
self.amp = amp
def get_acqmap(self, xin_is_list=True):
""" Return acqmap.
Inputs: xin_is_list True if input to acqmap is a list of xin """
# Potentially do acqmap setup here. Could include inference,
# cachining/computing quantities, instantiating objects used in acqmap
# definition. This becomes important when we do sequential opt of acqmaps.
return self.acqmap_list if xin_is_list else self.acqmap_single
def acqmap_list(self, xin_list):
""" Acqmap defined on a list of xin. """
def get_trans_data():
""" Returns transformed data. """
dt = DataTransformer(self.data.y, False)
return Namespace(X=self.data.X, y=dt.transform_data(self.data.y))
def apply_acq_to_pmlist(pmlist, acq_str, trans_data):
""" Apply acquisition to pmlist. """
acqp = Namespace(acq_str=acq_str, pmout_str='sample')
acq = Acquisitioner(trans_data, acqp, False)
acqfn = acq.acq_method
return [acqfn(p) for p in pmlist]
def georgegp_acqmap(acq_str):
""" Acqmaps for GeorgeGpPP """
trans_data = get_trans_data()
pp = GeorgeGpPP(trans_data, self.amp.modelp, False)
pmlist = pp.sample_pp_pred(self.amp.nppred, xin_list) if acq_str=='ts' \
else pp.sample_pp_post_pred(self.amp.nppred, xin_list)
return apply_acq_to_pmlist(pmlist, acq_str, trans_data)
def stangp_acqmap(acq_str):
""" Acqmaps for StanGpPP """
trans_data = get_trans_data()
pp = StanGpPP(trans_data, self.amp.modelp, False)
pp.infer_post_and_update_samples(print_result=True)
pmlist, _ = pp.sample_pp_pred(self.amp.nppred, xin_list) if acq_str=='ts' \
else pp.sample_pp_post_pred(self.amp.nppred, xin_list, full_cov=True, \
nloop=np.min([50,self.amp.nppred]))
return apply_acq_to_pmlist(pmlist, acq_str, trans_data)
def mygpdistmat_acqmap(acq_str):
""" Acqmaps for MyGpDistmatPP """
trans_data = get_trans_data()
pp = MyGpDistmatPP(trans_data, self.amp.modelp, False)
pp.infer_post_and_update_samples(print_result=True)
pmlist, _ = pp.sample_pp_pred(self.amp.nppred, xin_list) if acq_str=='ts' \
else pp.sample_pp_post_pred(self.amp.nppred, xin_list, full_cov=True)
return apply_acq_to_pmlist(pmlist, acq_str, trans_data)
# Mapping of am_str to acqmap
if self.amp.am_str=='georgegp_ei':
return georgegp_acqmap('ei')
elif self.amp.am_str=='georgegp_pi':
return georgegp_acqmap('pi')
elif self.amp.am_str=='georgegp_ucb':
return georgegp_acqmap('ucb')
elif self.amp.am_str=='georgegp_ts':
return georgegp_acqmap('ts')
elif self.amp.am_str=='stangp_ei':
return stangp_acqmap('ei')
elif self.amp.am_str=='stangp_pi':
return stangp_acqmap('pi')
elif self.amp.am_str=='stangp_ucb':
return stangp_acqmap('ucb')
elif self.amp.am_str=='stangp_ts':
return stangp_acqmap('ts')
elif self.amp.am_str=='mygpdistmat_ei':
return mygpdistmat_acqmap('ei')
elif self.amp.am_str=='mygpdistmat_pi':
return mygpdistmat_acqmap('pi')
elif self.amp.am_str=='mygpdistmat_ucb':
return mygpdistmat_acqmap('ucb')
elif self.amp.am_str=='mygpdistmat_ts':
return mygpdistmat_acqmap('ts')
elif self.amp.am_str=='null':
return [0. for xin in xin_list]
def acqmap_single(self, xin):
""" Acqmap defined on a single xin. Returns acqmap(xin) value, not list. """
return self.acqmap_list([xin])[0]
def print_str(self):
""" Print a description string """
print('*AcqMapper with amp='+str(self.amp)
+'.\n-----')
================================================
FILE: bo/acq/acqopt.py
================================================
"""
Classes to perform acquisition function optimization.
"""
from argparse import Namespace
import numpy as np
class AcqOptimizer(object):
""" Class to perform acquisition function optimization """
def __init__(self, optp=None, print_flag=True):
""" Constructor
Inputs:
optp - Namespace of opt parameters
print_flag - True or False
"""
self.set_opt_params(optp)
if print_flag: self.print_str()
def set_opt_params(self, optp):
""" Set the optimizer params.
Inputs:
acqp - Namespace of acquisition parameters """
if optp is None:
optp = Namespace(opt_str='rand', max_iter=1000)
self.optp = optp
def optimize(self, dom, am):
""" Optimize acqfn(probmap(x)) over x in domain """
if self.optp.opt_str=='rand':
return self.optimize_rand(dom, am)
def optimize_rand(self, dom, am):
""" Optimize acqmap(x) over domain via random search """
xin_list = dom.unif_rand_sample(self.optp.max_iter)
amlist = am.acqmap_list(xin_list)
return xin_list[np.argmin(amlist)]
# Utilities
def print_str(self):
""" print a description string """
print('*AcqOptimizer with optp='+str(self.optp)
+'.\n-----')
================================================
FILE: bo/acq/acquisition.py
================================================
"""
Classes to manage acquisition functions.
"""
from argparse import Namespace
import numpy as np
from scipy.stats import norm
class Acquisitioner(object):
""" Class to manage acquisition functions """
def __init__(self, data, acqp=None, print_flag=True):
""" Constructor
Parameters:
acqp - Namespace of acquisition parameters
print_flag - True or False
"""
self.data = data
self.set_acq_params(acqp)
self.set_acq_method()
if print_flag: self.print_str()
def set_acq_params(self, acqp):
""" Set the acquisition params.
Parameters:
acqp - Namespace of acquisition parameters """
if acqp is None:
acqp = Namespace(acq_str='ei', pmout_str='sample')
self.acqp = acqp
def set_acq_method(self):
""" Set the acquisition method """
if self.acqp.acq_str=='ei': self.acq_method = self.ei
if self.acqp.acq_str=='pi': self.acq_method = self.pi
if self.acqp.acq_str=='ts': self.acq_method = self.ts
if self.acqp.acq_str=='ucb': self.acq_method = self.ucb
if self.acqp.acq_str=='rand': self.acq_method = self.rand
if self.acqp.acq_str=='null': self.acq_method = self.null
#if self.acqp.acqStr=='map': return self.map
def ei(self, pmout):
""" Expected improvement (EI) """
if self.acqp.pmout_str=='sample':
return self.bbacq_ei(pmout)
def pi(self, pmout):
""" Probability of improvement (PI) """
if self.acqp.pmout_str=='sample':
return self.bbacq_pi(pmout)
def ucb(self, pmout):
""" Upper (lower) confidence bound (UCB) """
if self.acqp.pmout_str=='sample':
return self.bbacq_ucb(pmout)
def ts(self, pmout):
""" Thompson sampling (TS) """
if self.acqp.pmout_str=='sample':
return self.bbacq_ts(pmout)
def rand(self, pmout):
""" Uniform random sampling """
return np.random.random()
def null(self, pmout):
""" Return constant 0. """
return 0.
# Black Box Acquisition Functions
def bbacq_ei(self, pmout_samp, normal=False):
""" Black box acquisition: BB-EI
Input: pmout_samp: post-pred samples - np array (shape=(nsamp,1))
Returns: EI acq value """
youts = np.array(pmout_samp).flatten()
nsamp = youts.shape[0]
if normal:
mu = np.mean(youts)
sig = np.std(youts)
gam = (self.data.y.min() - mu) / sig
eiVal = -1*sig*(gam*norm.cdf(gam) + norm.pdf(gam))
else:
diffs = self.data.y.min() - youts
ind_below_min = np.argwhere(diffs>0)
eiVal = -1*np.sum(diffs[ind_below_min])/float(nsamp) if \
len(ind_below_min)>0 else 0
return eiVal
def bbacq_pi(self, pmout_samp, normal=False):
""" Black box acquisition: BB-PI
Input: pmout_samp: post-pred samples - np array (shape=(nsamp,1))
Returns: PI acq value """
youts = np.array(pmout_samp).flatten()
nsamp = youts.shape[0]
if normal:
mu = np.mean(youts)
sig = np.sqrt(np.var(youts))
piVal = -1*norm.cdf(self.data.y.min(),loc=mu,scale=sig)
else:
piVal = -1*len(np.argwhere(youts=self.domp.min_max[i][0] and
pt[i]<=self.domp.min_max[i][1] for i in range(self.ndimx)]
ret=False if False in bool_list else True
return ret
def unif_rand_sample(self, n=1):
""" Draws a sample uniformly at random from domain """
li = [np.random.uniform(mm[0], mm[1], n) for mm in self.domp.min_max]
return np.array(li).T
def print_str(self):
""" Print a description string """
print('*RealDomain with domp = ' + str(self.domp) + '.')
print('-----')
================================================
FILE: bo/ds/__init__.py
================================================
"""
Code for makept (serializing and subprocesses) strategy.
"""
================================================
FILE: bo/ds/makept.py
================================================
"""
Make a point in a domain, and serialize it.
"""
import sys
import os
sys.path.append(os.path.expanduser('./'))
from argparse import Namespace, ArgumentParser
import pickle
import time
import numpy as np
from bo.dom.real import RealDomain
from bo.dom.list import ListDomain
from bo.acq.acqmap import AcqMapper
from bo.acq.acqopt import AcqOptimizer
def main(args, search_space, printinfo=False):
starttime = time.time()
# Load config and data
makerp = pickle.load(open(args.configpkl, 'rb'))
data = pickle.load(open(args.datapkl, 'rb'))
if hasattr(args, 'mode') and args.mode == 'single_process':
makerp.domp.mode = args.mode
makerp.domp.iteridx = args.iteridx
makerp.amp.modelp.mode = args.mode
else:
np.random.seed(args.seed)
# Instantiate Domain, AcqMapper, AcqOptimizer
dom = get_domain(makerp.domp, search_space)
am = AcqMapper(data, makerp.amp, False)
ao = AcqOptimizer(makerp.optp, False)
# Optimize over domain to get nextpt
nextpt = ao.optimize(dom, am)
# Serialize nextpt
with open(args.nextptpkl, 'wb') as f:
pickle.dump(nextpt, f)
# Print
itertime = time.time()-starttime
if printinfo: print_info(nextpt, itertime, args.nextptpkl)
def get_domain(domp, search_space):
""" Return Domain object. """
if not hasattr(domp, 'dom_str'):
domp.dom_str = 'real'
if domp.dom_str=='real':
return RealDomain(domp, False)
elif domp.dom_str=='list':
return ListDomain(search_space, domp, False)
def print_info(nextpt, itertime, nextptpkl):
print('*Found nextpt = ' + str(nextpt) + '.')
print('*Saved nextpt as ' + nextptpkl + '.')
print('*Timing: makept took ' + str(itertime) + ' seconds.')
print('-----')
if __name__ == "__main__":
parser = ArgumentParser(description='Args for a single instance of acquisition optimization.')
parser.add_argument('--seed', dest='seed', type=int, default=1111)
parser.add_argument('--configpkl', dest='configpkl', type=str, default='config.pkl')
parser.add_argument('--datapkl', dest='datapkl', type=str, default='data.pkl')
parser.add_argument('--nextptpkl', dest='nextptpkl', type=str, default='nextpt.pkl')
args = parser.parse_args()
main(args, printinfo=False)
================================================
FILE: bo/fn/__init__.py
================================================
"""
Code for synthetic functions to query (perform experiment on).
"""
================================================
FILE: bo/fn/functionhandler.py
================================================
"""
Classes to handle functions.
"""
from argparse import Namespace
import numpy as np
def get_fh(fn, data=None, fhp=None, print_flag=True):
""" Returns a function handler object """
if fhp is None:
fhp=Namespace(fhstr='basic', namestr='noname')
# Return FH object
if fhp.fhstr=='basic':
return BasicFH(fn, data, fhp, print_flag)
elif fhp.fhstr=='extrainfo':
return ExtraInfoFH(fn, data, fhp, print_flag)
elif fhp.fhstr=='nannn':
return NanNNFH(fn, data, fhp, print_flag)
elif fhp.fhstr=='replacenannn':
return ReplaceNanNNFH(fn, data, fhp, print_flag)
elif fhp.fhstr=='object':
return ObjectFH(fn, data, fhp, print_flag)
class BasicFH(object):
""" Class to handle basic functions, which map from an array xin to a real
value yout. """
def __init__(self, fn, data=None, fhp=None, print_flag=True):
""" Constructor.
Inputs:
pmp - Namespace of probmap params
print_flag - True or False
"""
self.fn = fn
self.data = data
self.fhp = fhp
if print_flag: self.print_str()
def call_fn_and_add_data(self, xin):
""" Call self.fn(xin), and update self.data """
yout = self.fn(xin)
print('new datapoint score', yout)
self.add_data_single(xin, yout)
def add_data_single(self, xin, yout):
""" Update self.data with a single xin yout pair.
Inputs:
xin: np.array size=(1, -1)
yout: np.array size=(1, 1) """
xin = np.array(xin).reshape(1, -1)
yout = np.array(yout).reshape(1, 1)
newdata = Namespace(X=xin, y=yout)
self.add_data(newdata)
def add_data(self, newdata):
""" Update self.data with newdata Namespace.
Inputs:
newdata: Namespace with fields X and y """
if self.data is None:
self.data = newdata
else:
self.data.X = np.concatenate((self.data.X, newdata.X), 0)
self.data.y = np.concatenate((self.data.y, newdata.y), 0)
def print_str(self):
""" Print a description string. """
print('*BasicFH with fhp='+str(self.fhp)
+'.\n-----')
class ExtraInfoFH(BasicFH):
""" Class to handle functions that map from an array xin to a real
value yout, but also return extra info """
def __init__(self, fn, data=None, fhp=None, print_flag=True):
""" Constructor.
Inputs:
pmp - Namespace of probmap params
print_flag - True or False
"""
super(ExtraInfoFH, self).__init__(fn, data, fhp, False)
self.extrainfo = []
if print_flag: self.print_str()
def call_fn_and_add_data(self, xin):
""" Call self.fn(xin), and update self.data """
yout, exinf = self.fn(xin)
self.add_data_single(xin, yout)
self.extrainfo.append(exinf)
def print_str(self):
""" Print a description string. """
print('*ExtraInfoFH with fhp='+str(self.fhp)
+'.\n-----')
class NanNNFH(BasicFH):
""" Class to handle NN functions that map from an array xin to either
a real value yout or np.NaN, but also return extra info """
def __init__(self, fn, data=None, fhp=None, print_flag=True):
""" Constructor.
Inputs:
pmp - Namespace of probmap params
print_flag - True or False
"""
super(NanNNFH, self).__init__(fn, data, fhp, False)
self.extrainfo = []
if print_flag: self.print_str()
def call_fn_and_add_data(self, xin):
""" Call self.fn(xin), and update self.data """
timethresh = 60.
yout, walltime = self.fn(xin)
if walltime > timethresh:
self.add_data_single_nan(xin)
else:
self.add_data_single(xin, yout)
self.possibly_init_xnan()
exinf = Namespace(xin=xin, yout=yout, walltime=walltime)
self.extrainfo.append(exinf)
def add_data_single_nan(self, xin):
""" Update self.data.X_nan with a single xin.
Inputs:
xin: np.array size=(1, -1) """
xin = xin.reshape(1,-1)
newdata = Namespace(X = np.ones((0, xin.shape[1])),
y = np.ones((0, 1)),
X_nan = xin)
self.add_data_nan(newdata)
def add_data_nan(self, newdata):
""" Update self.data with newdata Namespace.
Inputs:
newdata: Namespace with fields X, y, X_nan """
if self.data is None:
self.data = newdata
else:
self.data.X_nan = np.concatenate((self.data.X_nan, newdata.X_nan), 0)
def possibly_init_xnan(self):
""" If self.data doesn't have X_nan, then create it. """
if not hasattr(self.data, 'X_nan'):
self.data.X_nan = np.ones((0, self.data.X.shape[1]))
def print_str(self):
""" Print a description string. """
print('*NanNNFH with fhp='+str(self.fhp)
+'.\n-----')
class ReplaceNanNNFH(BasicFH):
""" Class to handle NN functions that map from an array xin to either
a real value yout or np.NaN. If np.NaN, we replace it with a large
positive value. We also return extra info """
def __init__(self, fn, data=None, fhp=None, print_flag=True):
""" Constructor.
Inputs:
pmp - Namespace of probmap params
print_flag - True or False
"""
super(ReplaceNanNNFH, self).__init__(fn, data, fhp, False)
self.extrainfo = []
if print_flag: self.print_str()
def call_fn_and_add_data(self, xin):
""" Call self.fn(xin), and update self.data """
timethresh = 60.
replace_nan_val = 5.
yout, walltime = self.fn(xin)
if walltime > timethresh:
yout = replace_nan_val
self.add_data_single(xin, yout)
exinf = Namespace(xin=xin, yout=yout, walltime=walltime)
self.extrainfo.append(exinf)
def print_str(self):
""" Print a description string. """
print('*ReplaceNanNNFH with fhp='+str(self.fhp)
+'.\n-----')
class ObjectFH(object):
""" Class to handle basic functions, which map from some object xin to a real
value yout. """
def __init__(self, fn, data=None, fhp=None, print_flag=True):
""" Constructor.
Inputs:
pmp - Namespace of probmap params
print_flag - True or False
"""
self.fn = fn
self.data = data
self.fhp = fhp
if print_flag: self.print_str()
def call_fn_and_add_data(self, xin):
""" Call self.fn(xin), and update self.data """
yout = self.fn(xin)
self.add_data_single(xin, yout)
def add_data_single(self, xin, yout):
""" Update self.data with a single xin yout pair. """
newdata = Namespace(X=[xin], y=np.array(yout).reshape(1, 1))
self.add_data(newdata)
def add_data(self, newdata):
""" Update self.data with newdata Namespace.
Inputs:
newdata: Namespace with fields X and y """
if self.data is None:
self.data = newdata
else:
self.data.X.extend(newdata.X)
self.data.y = np.concatenate((self.data.y, newdata.y), 0)
def print_str(self):
""" Print a description string. """
print('*ObjectFH with fhp='+str(self.fhp)
+'.\n-----')
================================================
FILE: bo/pp/__init__.py
================================================
"""
Code for defining and running probabilistic programs.
"""
================================================
FILE: bo/pp/gp/__init__.py
================================================
"""
Code for Gaussian process (GP) utilities and functions.
"""
================================================
FILE: bo/pp/gp/gp_utils.py
================================================
"""
Utilities for Gaussian process (GP) inference
"""
import numpy as np
from scipy.linalg import solve_triangular
from scipy.spatial.distance import cdist
#import GPy as gpy
def kern_gibbscontext(xmatcon1, xmatcon2, xmatact1, xmatact2, theta, alpha,
lscon, whichlsfn=1):
""" Gibbs kernel (ls_fn of context only) """
actdim = xmatact1.shape[1]
lsarr1 = ls_fn(xmatcon1, theta, whichlsfn).flatten()
lsarr2 = ls_fn(xmatcon2, theta, whichlsfn).flatten()
sum_sq_ls = np.add.outer(lsarr1, lsarr2)
inexp = -1. * np.divide(cdist(xmatact1, xmatact2, 'sqeuclidean'), sum_sq_ls)
prod_ls = np.outer(lsarr1, lsarr2)
#coef = np.power(np.divide(2*prod_ls, sum_sq_ls), actdim/2.) # Correct
coef = 1.
kern_gibbscontext_only_ns = np.multiply(coef, np.exp(inexp))
kern_expquad_ns = kern_exp_quad_noscale(xmatcon1, xmatcon2, lscon)
return alpha**2 * np.multiply(kern_gibbscontext_only_ns, kern_expquad_ns)
def kern_gibbs1d(xmat1, xmat2, theta, alpha):
""" Gibbs kernel in 1d """
lsarr1 = ls_fn(xmat1, theta).flatten()
lsarr2 = ls_fn(xmat2, theta).flatten()
sum_sq_ls = np.add.outer(lsarr1, lsarr2)
prod_ls = np.outer(lsarr1, lsarr2) #TODO product of this for each dim
coef = np.sqrt(np.divide(2*prod_ls, sum_sq_ls))
inexp = cdist(xmat1, xmat2, 'sqeuclidean') / sum_sq_ls #TODO sum of this for each dim
return alpha**2 * coef * np.exp(-1 * inexp)
def ls_fn(xmat, theta, whichlsfn=1):
theta = np.array(theta).reshape(-1,1)
if theta.shape[0]==2:
if whichlsfn==1 or whichlsfn==2:
return np.log(1 + np.exp(theta[0][0] + np.matmul(xmat,theta[1]))) # softplus transform
elif whichlsfn==3:
return np.exp(theta[0][0] + np.matmul(xmat,theta[1])) # exp transform
elif theta.shape[0]==3:
if whichlsfn==1:
return np.log(1 + np.exp(theta[0][0] + np.matmul(xmat,theta[1]) +
np.matmul(np.power(xmat,2),theta[2]))) # softplus transform
elif whichlsfn==2:
return np.log(1 + np.exp(theta[0][0] + np.matmul(xmat,theta[1]) +
np.matmul(np.abs(xmat),theta[2]))) # softplus on abs transform
elif whichlsfn==3:
return np.exp(theta[0][0] + np.matmul(xmat,theta[1]) +
np.matmul(np.power(xmat,2),theta[2])) # exp transform
else:
print('ERROR: theta parameter is incorrect.')
def kern_matern32(xmat1, xmat2, ls, alpha):
""" Matern 3/2 kernel, currently using GPy """
kern = gpy.kern.Matern32(input_dim=xmat1.shape[1], variance=alpha**2,
lengthscale=ls)
return kern.K(xmat1,xmat2)
def kern_exp_quad(xmat1, xmat2, ls, alpha):
""" Exponentiated quadratic kernel function aka squared exponential kernel
aka RBF kernel """
return alpha**2 * kern_exp_quad_noscale(xmat1, xmat2, ls)
def kern_exp_quad_noscale(xmat1, xmat2, ls):
""" Exponentiated quadratic kernel function aka squared exponential kernel
aka RBF kernel, without scale parameter. """
sq_norm = (-1/(2 * ls**2)) * cdist(xmat1, xmat2, 'sqeuclidean')
return np.exp(sq_norm)
def squared_euc_distmat(xmat1, xmat2, coef=1.):
""" Distance matrix of squared euclidean distance (multiplied by coef)
between points in xmat1 and xmat2. """
return coef * cdist(xmat1, xmat2, 'sqeuclidean')
def kern_distmat(xmat1, xmat2, ls, alpha, distfn):
""" Kernel for a given distmat, via passed-in distfn (which is assumed to be
fn of xmat1 and xmat2 only) """
distmat = distfn(xmat1, xmat2)
sq_norm = -distmat / ls**2
return alpha**2 * np.exp(sq_norm)
def get_cholesky_decomp(k11_nonoise, sigma, psd_str):
""" Returns cholesky decomposition """
if psd_str == 'try_first':
k11 = k11_nonoise + sigma**2 * np.eye(k11_nonoise.shape[0])
try:
return stable_cholesky(k11, False)
except np.linalg.linalg.LinAlgError:
return get_cholesky_decomp(k11_nonoise, sigma, 'project_first')
elif psd_str == 'project_first':
k11_nonoise = project_symmetric_to_psd_cone(k11_nonoise)
return get_cholesky_decomp(k11_nonoise, sigma, 'is_psd')
elif psd_str == 'is_psd':
k11 = k11_nonoise + sigma**2 * np.eye(k11_nonoise.shape[0])
return stable_cholesky(k11)
def stable_cholesky(mmat, make_psd=True):
""" Returns a 'stable' cholesky decomposition of mmat """
if mmat.size == 0:
return mmat
try:
lmat = np.linalg.cholesky(mmat)
except np.linalg.linalg.LinAlgError as e:
if not make_psd:
raise e
diag_noise_power = -11
max_mmat = np.diag(mmat).max()
diag_noise = np.diag(mmat).max() * 1e-11
break_loop = False
while not break_loop:
try:
lmat = np.linalg.cholesky(mmat + ((10**diag_noise_power) * max_mmat) *
np.eye(mmat.shape[0]))
break_loop = True
except np.linalg.linalg.LinAlgError:
if diag_noise_power > -9:
print('stable_cholesky failed with diag_noise_power=%d.'%(diag_noise_power))
diag_noise_power += 1
if diag_noise_power >= 5:
print('***** stable_cholesky failed: added diag noise = %e'%(diag_noise))
return lmat
def project_symmetric_to_psd_cone(mmat, is_symmetric=True, epsilon=0):
""" Project symmetric matrix mmat to the PSD cone """
if is_symmetric:
try:
eigvals, eigvecs = np.linalg.eigh(mmat)
except np.linalg.LinAlgError:
print('LinAlgError encountered with np.eigh. Defaulting to eig.')
eigvals, eigvecs = np.linalg.eig(mmat)
eigvals = np.real(eigvals)
eigvecs = np.real(eigvecs)
else:
eigvals, eigvecs = np.linalg.eig(mmat)
clipped_eigvals = np.clip(eigvals, epsilon, np.inf)
return (eigvecs * clipped_eigvals).dot(eigvecs.T)
def solve_lower_triangular(amat, b):
""" Solves amat*x=b when amat is lower triangular """
return solve_triangular_base(amat, b, lower=True)
def solve_upper_triangular(amat, b):
""" Solves amat*x=b when amat is upper triangular """
return solve_triangular_base(amat, b, lower=False)
def solve_triangular_base(amat, b, lower):
""" Solves amat*x=b when amat is a triangular matrix. """
if amat.size == 0 and b.shape[0] == 0:
return np.zeros((b.shape))
else:
return solve_triangular(amat, b, lower=lower)
def sample_mvn(mu, covmat, nsamp):
""" Sample from multivariate normal distribution with mean mu and covariance
matrix covmat """
mu = mu.reshape(-1,)
ndim = len(mu)
lmat = stable_cholesky(covmat)
umat = np.random.normal(size=(ndim, nsamp))
return lmat.dot(umat).T + mu
================================================
FILE: bo/pp/pp_core.py
================================================
"""
Base classes for probabilistic programs.
"""
import pickle
class DiscPP(object):
""" Parent class for discriminative probabilistic programs """
def __init__(self):
""" Constructor """
self.sample_list = []
if not hasattr(self,'data'):
raise NotImplementedError('Implement var data in a child class')
#if not hasattr(self,'ndimx'):
#raise NotImplementedError('Implement var ndimx in a child class')
#if not hasattr(self,'ndataInit'):
#raise NotImplementedError('Implement var ndataInit in a child class')
def infer_post_and_update_samples(self,nsamp):
""" Run an inference algorithm (given self.data), draw samples from the
posterior, and store in self.sample_list. """
raise NotImplementedError('Implement method in a child class')
def sample_pp_post_pred(self,nsamp,input_list):
""" Sample nsamp times from PP posterior predictive, for each x-input in
input_list """
raise NotImplementedError('Implement method in a child class')
def sample_pp_pred(self,nsamp,input_list,lv_list=None):
""" Sample nsamp times from PP predictive for parameter lv, for each
x-input in input_list. If lv is None, draw it uniformly at random
from self.sample_list. """
raise NotImplementedError('Implement method in a child class')
def add_new_data(self,newData):
""" Add data (newData) to self.data """
raise NotImplementedError('Implement method in a child class')
def get_namespace_to_save(self):
""" Return namespace containing object info (to save to file) """
raise NotImplementedError('Implement method in a child class')
def save_namespace_to_file(self,fileStr,printFlag):
""" Saves results from get_namespace_to_save in fileStr """
ppNamespaceToSave = self.get_namespace_to_save()
ff = open(fileStr,'wb')
pickle.dump(ppNamespaceToSave,ff)
ff.close()
if printFlag:
print('*Saved DiscPP Namespace in pickle file: ' +fileStr+'\n-----')
================================================
FILE: bo/pp/pp_gp_george.py
================================================
"""
Classes for hierarchical GP models with George PP
"""
from argparse import Namespace
import numpy as np
import scipy.optimize as spo
import george
import emcee
from bo.pp.pp_core import DiscPP
class GeorgeGpPP(DiscPP):
""" Hierarchical GPs implemented with George """
def __init__(self,data=None,modelp=None,printFlag=True):
""" Constructor """
self.set_data(data)
self.set_model_params(modelp)
self.ndimx = self.modelp.ndimx
self.set_kernel()
self.set_model()
super(GeorgeGpPP,self).__init__()
if printFlag:
self.print_str()
def set_data(self,data):
if data is None:
pass #TODO: handle case where there's no data
self.data = data
def set_model_params(self,modelp):
if modelp is None:
modelp = Namespace(ndimx=1, noiseVar=1e-3, kernLs=1.5, kernStr='mat',
fitType='mle')
self.modelp = modelp
def set_kernel(self):
""" Set kernel for GP """
if self.modelp.kernStr=='mat':
self.kernel = self.data.y.var() * \
george.kernels.Matern52Kernel(self.modelp.kernLs, ndim=self.ndimx)
if self.modelp.kernStr=='rbf': # NOTE: periodically produces errors
self.kernel = self.data.y.var() * \
george.kernels.ExpSquaredKernel(self.modelp.kernLs, ndim=self.ndimx)
def set_model(self):
""" Set GP regression model """
self.model = self.get_model()
self.model.compute(self.data.X)
self.fit_hyperparams(printOut=False)
def get_model(self):
""" Returns GPRegression model """
return george.GP(kernel=self.kernel,fit_mean=True)
def fit_hyperparams(self,printOut=False):
if self.modelp.fitType=='mle':
spo.minimize(self.neg_log_like, self.model.get_parameter_vector(),
jac=True)
elif self.modelp.fitType=='bayes':
self.nburnin = 200
nsamp = 200
nwalkers = 36
gpdim = len(self.model)
self.sampler = emcee.EnsembleSampler(nwalkers, gpdim, self.log_post)
p0 = self.model.get_parameter_vector() + 1e-4*np.random.randn(nwalkers,
gpdim)
print 'Running burn-in.'
p0, _, _ = self.sampler.run_mcmc(p0, self.nburnin)
print 'Running main chain.'
self.sampler.run_mcmc(p0, nsamp)
if printOut:
print 'Final GP hyperparam (in opt or MCMC chain):'
print self.model.get_parameter_dict()
def infer_post_and_update_samples(self):
""" Update self.sample_list """
self.sample_list = [None] #TODO: need to not-break ts fn in maker_bayesopt.py
def sample_pp_post_pred(self,nsamp,input_list):
""" Sample from posterior predictive of PP.
Inputs:
input_list - list of np arrays size=(-1,)
Returns:
list (len input_list) of np arrays (size=(nsamp,1))."""
inputArray = np.array(input_list)
if self.modelp.fitType=='mle':
inputArray = np.array(input_list)
ppredArray = self.model.sample_conditional(self.data.y.flatten(),
inputArray, nsamp).T
elif self.modelp.fitType=='bayes':
ppredArray = np.zeros(shape=[len(input_list),nsamp])
for s in range(nsamp):
walkidx = np.random.randint(self.sampler.chain.shape[0])
sampidx = np.random.randint(self.nburnin, self.sampler.chain.shape[1])
hparamSamp = self.sampler.chain[walkidx, sampidx]
print 'hparamSamp = ' + str(hparamSamp) # TODO: remove print statement
self.model.set_parameter_vector(hparamSamp)
ppredArray[:,s] = self.model.sample_conditional(self.data.y.flatten(),
inputArray, 1).flatten()
return list(ppredArray) # each element is row in ppredArray matrix
def sample_pp_pred(self,nsamp,input_list,lv=None):
""" Sample from predictive of PP for parameter lv.
Returns: list (len input_list) of np arrays (size (nsamp,1))."""
if self.modelp.fitType=='bayes':
print('*WARNING: fitType=bayes not implemented for sample_pp_pred. \
Reverting to fitType=mle')
# TODO: Equivalent algo for fitType=='bayes':
# - draw posterior sample path over all xin in input_list
# - draw pred samples around sample path pt, based on noise model
inputArray = np.array(input_list)
samplePath = self.model.sample_conditional(self.data.y.flatten(),
inputArray).reshape(-1,)
return [np.random.normal(s,np.sqrt(self.modelp.noiseVar),nsamp).reshape(-1,)
for s in samplePath]
def neg_log_like(self,hparams):
""" Compute and return the negative log likelihood for model
hyperparameters hparams, as well as its gradient. """
self.model.set_parameter_vector(hparams)
g = self.model.grad_log_likelihood(self.data.y.flatten(), quiet=True)
return -self.model.log_likelihood(self.data.y.flatten(), quiet=True), -g
def log_post(self,hparams):
""" Compute and return the log posterior density (up to constant of
proportionality) for the model hyperparameters hparams. """
# Uniform prior between -100 and 100, for each hyperparam
if np.any((-100 > hparams[1:]) + (hparams[1:] > 100)):
return -np.inf
self.model.set_parameter_vector(hparams)
return self.model.log_likelihood(self.data.y.flatten(), quiet=True)
# Utilities
def print_str(self):
""" Print a description string """
print '*GeorgeGpPP with modelp='+str(self.modelp)+'.'
print '-----'
================================================
FILE: bo/pp/pp_gp_my_distmat.py
================================================
"""
Classes for GP models without any PP backend, using a given distance matrix.
"""
from argparse import Namespace
import time
import copy
import numpy as np
from scipy.spatial.distance import cdist
from bo.pp.pp_core import DiscPP
from bo.pp.gp.gp_utils import kern_exp_quad, kern_matern32, \
get_cholesky_decomp, solve_upper_triangular, solve_lower_triangular, \
sample_mvn, squared_euc_distmat, kern_distmat
from bo.util.print_utils import suppress_stdout_stderr
class MyGpDistmatPP(DiscPP):
""" GPs using a kernel specified by a given distance matrix, without any PP
backend """
def __init__(self, data=None, modelp=None, printFlag=True):
""" Constructor """
self.set_model_params(modelp)
self.set_data(data)
self.set_model()
super(MyGpDistmatPP,self).__init__()
if printFlag:
self.print_str()
def set_model_params(self, modelp):
""" Set self.modelp """
if modelp is None:
pass #TODO
self.modelp = modelp
def set_data(self, data):
""" Set self.data """
if data is None:
pass #TODO
self.data_init = copy.deepcopy(data)
self.data = copy.deepcopy(self.data_init)
def set_model(self):
""" Set GP regression model """
self.model = self.get_model()
def get_model(self):
""" Returns model object """
return None
def infer_post_and_update_samples(self, print_result=False):
""" Update self.sample_list """
self.sample_list = [Namespace(ls=self.modelp.kernp.ls,
alpha=self.modelp.kernp.alpha,
sigma=self.modelp.kernp.sigma)]
if print_result: self.print_inference_result()
def get_distmat(self, xmat1, xmat2):
""" Get distance matrix """
#return squared_euc_distmat(xmat1, xmat2, .5)
from data import Data
self.distmat = Data.generate_distance_matrix
#print('distmat')
#print(self.distmat(xmat1, xmat2, self.modelp.distance))
return self.distmat(xmat1, xmat2, self.modelp.distance)
def print_inference_result(self):
""" Print results of stan inference """
print('*ls pt est = '+str(self.sample_list[0].ls)+'.')
print('*alpha pt est = '+str(self.sample_list[0].alpha)+'.')
print('*sigma pt est = '+str(self.sample_list[0].sigma)+'.')
print('-----')
def sample_pp_post_pred(self, nsamp, input_list, full_cov=False):
""" Sample from posterior predictive of PP.
Inputs:
input_list - list of np arrays size=(-1,)
Returns:
list (len input_list) of np arrays (size=(nsamp,1))."""
samp = self.sample_list[0]
postmu, postcov = self.gp_post(self.data.X, self.data.y, input_list,
samp.ls, samp.alpha, samp.sigma, full_cov)
if full_cov:
ppred_list = list(sample_mvn(postmu, postcov, nsamp))
else:
ppred_list = list(np.random.normal(postmu.reshape(-1,),
postcov.reshape(-1,),
size=(nsamp, len(input_list))))
return list(np.stack(ppred_list).T), ppred_list
def sample_pp_pred(self, nsamp, input_list, lv=None):
""" Sample from predictive of PP for parameter lv.
Returns: list (len input_list) of np arrays (size (nsamp,1))."""
if lv is None:
lv = self.sample_list[0]
postmu, postcov = self.gp_post(self.data.X, self.data.y, input_list, lv.ls,
lv.alpha, lv.sigma)
pred_list = list(sample_mvn(postmu, postcov, 1)) ###TODO: sample from this mean nsamp times
return list(np.stack(pred_list).T), pred_list
def gp_post(self, x_train_list, y_train_arr, x_pred_list, ls, alpha, sigma,
full_cov=True):
""" Compute parameters of GP posterior """
kernel = lambda a, b, c, d: kern_distmat(a, b, c, d, self.get_distmat)
k11_nonoise = kernel(x_train_list, x_train_list, ls, alpha)
lmat = get_cholesky_decomp(k11_nonoise, sigma, 'try_first')
smat = solve_upper_triangular(lmat.T, solve_lower_triangular(lmat,
y_train_arr))
k21 = kernel(x_pred_list, x_train_list, ls, alpha)
mu2 = k21.dot(smat)
k22 = kernel(x_pred_list, x_pred_list, ls, alpha)
vmat = solve_lower_triangular(lmat, k21.T)
k2 = k22 - vmat.T.dot(vmat)
if full_cov is False:
k2 = np.sqrt(np.diag(k2))
return mu2, k2
# Utilities
def print_str(self):
""" Print a description string """
print('*MyGpDistmatPP with modelp='+str(self.modelp)+'.')
print('-----')
================================================
FILE: bo/pp/pp_gp_stan.py
================================================
"""
Classes for GP models with Stan
"""
from argparse import Namespace
import time
import numpy as np
import copy
from bo.pp.pp_core import DiscPP
import bo.pp.stan.gp_hier2 as gpstan2
import bo.pp.stan.gp_hier3 as gpstan3
import bo.pp.stan.gp_hier2_matern as gpstan2_matern
from bo.pp.gp.gp_utils import kern_exp_quad, kern_matern32, \
get_cholesky_decomp, solve_upper_triangular, solve_lower_triangular, \
sample_mvn
from bo.util.print_utils import suppress_stdout_stderr
class StanGpPP(DiscPP):
""" Hierarchical GPs implemented with Stan """
def __init__(self, data=None, modelp=None, printFlag=True):
""" Constructor """
self.set_model_params(modelp)
self.set_data(data)
self.ndimx = self.modelp.ndimx
self.set_model()
super(StanGpPP,self).__init__()
if printFlag:
self.print_str()
def set_model_params(self,modelp):
if modelp is None:
modelp = Namespace(ndimx=1, model_str='optfixedsig',
gp_mean_transf_str='constant')
if modelp.model_str=='optfixedsig':
modelp.kernp = Namespace(u1=.1, u2=5., n1=10., n2=10., sigma=1e-5)
modelp.infp = Namespace(niter=1000)
elif modelp.model_str=='opt' or modelp.model_str=='optmatern32':
modelp.kernp = Namespace(ig1=1., ig2=5., n1=10., n2=20., n3=.01,
n4=.01)
modelp.infp = Namespace(niter=1000)
elif modelp.model_str=='samp' or modelp.model_str=='sampmatern32':
modelp.kernp = Namespace(ig1=1., ig2=5., n1=10., n2=20., n3=.01,
n4=.01)
modelp.infp = Namespace(niter=1500, nwarmup=500)
self.modelp = modelp
def set_data(self, data):
""" Set self.data """
if data is None:
pass #TODO: handle case where there's no data
self.data_init = copy.deepcopy(data)
self.data = self.get_transformed_data(self.data_init,
self.modelp.gp_mean_transf_str)
def get_transformed_data(self, data, transf_str='linear'):
""" Transform data, for non-zero-mean GP """
newdata = Namespace(X=data.X)
if transf_str=='linear':
mmat,_,_,_ = np.linalg.lstsq(np.concatenate([data.X,
np.ones((data.X.shape[0],1))],1), data.y.flatten(), rcond=None)
self.gp_mean_vec = lambda x: np.matmul(np.concatenate([x,
np.ones((x.shape[0],1))],1), mmat)
newdata.y = data.y - self.gp_mean_vec(data.X).reshape(-1,1)
if transf_str=='constant':
yconstant = data.y.mean()
#yconstant = 0.
self.gp_mean_vec = lambda x: np.array([yconstant for xcomp in x])
newdata.y = data.y - self.gp_mean_vec(data.X).reshape(-1,1)
return newdata
def set_model(self):
""" Set GP regression model """
self.model = self.get_model()
def get_model(self):
""" Returns GPRegression model """
if self.modelp.model_str=='optfixedsig':
return gpstan3.get_model(print_status=False)
elif self.modelp.model_str=='opt' or self.modelp.model_str=='samp':
return gpstan2.get_model(print_status=False)
elif self.modelp.model_str=='optmatern32' or \
self.modelp.model_str=='sampmatern32':
return gpstan2_matern.get_model(print_status=False)
def infer_post_and_update_samples(self, seed=5000012, print_result=False):
""" Update self.sample_list """
data_dict = self.get_stan_data_dict()
with suppress_stdout_stderr():
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' \
or self.modelp.model_str=='optmatern32':
stanout = self.model.optimizing(data_dict, iter=self.modelp.infp.niter,
#seed=seed, as_vector=True, algorithm='Newton')
seed=seed, as_vector=True, algorithm='LBFGS')
elif self.modelp.model_str=='samp' or self.modelp.model_str=='sampmatern32':
stanout = self.model.sampling(data_dict, iter=self.modelp.infp.niter +
self.modelp.infp.nwarmup, warmup=self.modelp.infp.nwarmup, chains=1,
seed=seed, refresh=1000)
print('-----')
self.sample_list = self.get_sample_list_from_stan_out(stanout)
if print_result: self.print_inference_result()
def get_stan_data_dict(self):
""" Return data dict for stan sampling method """
if self.modelp.model_str=='optfixedsig':
return {'u1':self.modelp.kernp.u1, 'u2':self.modelp.kernp.u2,
'n1':self.modelp.kernp.n1, 'n2':self.modelp.kernp.n2,
'sigma':self.modelp.kernp.sigma, 'D':self.ndimx,
'N':len(self.data.X), 'x':self.data.X, 'y':self.data.y.flatten()}
elif self.modelp.model_str=='opt' or self.modelp.model_str=='samp':
return {'ig1':self.modelp.kernp.ig1, 'ig2':self.modelp.kernp.ig2,
'n1':self.modelp.kernp.n1, 'n2':self.modelp.kernp.n2,
'n3':self.modelp.kernp.n3, 'n4':self.modelp.kernp.n4,
'D':self.ndimx, 'N':len(self.data.X), 'x':self.data.X,
'y':self.data.y.flatten()}
elif self.modelp.model_str=='optmatern32' or \
self.modelp.model_str=='sampmatern32':
return {'ig1':self.modelp.kernp.ig1, 'ig2':self.modelp.kernp.ig2,
'n1':self.modelp.kernp.n1, 'n2':self.modelp.kernp.n2,
'n3':self.modelp.kernp.n3, 'n4':self.modelp.kernp.n4,
'D':self.ndimx, 'N':len(self.data.X), 'x':self.data.X,
'y':self.data.y.flatten(), 'covid':2}
def get_sample_list_from_stan_out(self, stanout):
""" Convert stan output to sample_list """
if self.modelp.model_str=='optfixedsig':
return [Namespace(ls=stanout['rho'], alpha=stanout['alpha'],
sigma=self.modelp.kernp.sigma)]
elif self.modelp.model_str=='opt' or self.modelp.model_str=='optmatern32':
return [Namespace(ls=stanout['rho'], alpha=stanout['alpha'],
sigma=stanout['sigma'])]
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampmatern32':
sdict = stanout.extract(['rho','alpha','sigma'])
return [Namespace(ls=sdict['rho'][i], alpha=sdict['alpha'][i],
sigma=sdict['sigma'][i]) for i in range(sdict['rho'].shape[0])]
def print_inference_result(self):
""" Print results of stan inference """
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' or \
self.modelp.model_str=='optmatern32':
print('*ls pt est = '+str(self.sample_list[0].ls)+'.')
print('*alpha pt est = '+str(self.sample_list[0].alpha)+'.')
print('*sigma pt est = '+str(self.sample_list[0].sigma)+'.')
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampmatern32':
ls_arr = np.array([ns.ls for ns in self.sample_list])
alpha_arr = np.array([ns.alpha for ns in self.sample_list])
sigma_arr = np.array([ns.sigma for ns in self.sample_list])
print('*ls mean = '+str(ls_arr.mean())+'.')
print('*ls std = '+str(ls_arr.std())+'.')
print('*alpha mean = '+str(alpha_arr.mean())+'.')
print('*alpha std = '+str(alpha_arr.std())+'.')
print('*sigma mean = '+str(sigma_arr.mean())+'.')
print('*sigma std = '+str(sigma_arr.std())+'.')
print('-----')
def sample_pp_post_pred(self, nsamp, input_list, full_cov=False, nloop=None):
""" Sample from posterior predictive of PP.
Inputs:
input_list - list of np arrays size=(-1,)
Returns:
list (len input_list) of np arrays (size=(nsamp,1))."""
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' or \
self.modelp.model_str=='optmatern32':
nloop = 1
sampids = [0]
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampmatern32':
if nloop is None: nloop=nsamp
nsamp = int(nsamp/nloop)
sampids = np.random.randint(len(self.sample_list), size=(nloop,))
ppred_list = []
for i in range(nloop):
samp = self.sample_list[sampids[i]]
postmu, postcov = self.gp_post(self.data.X, self.data.y,
np.stack(input_list), samp.ls, samp.alpha, samp.sigma, full_cov)
if full_cov:
ppred_list.extend(list(sample_mvn(postmu, postcov, nsamp)))
else:
ppred_list.extend(list(np.random.normal(postmu.reshape(-1,),
postcov.reshape(-1,), size=(nsamp, len(input_list)))))
return self.get_reverse_transform(list(np.stack(ppred_list).T), ppred_list,
input_list)
def sample_pp_pred(self, nsamp, input_list, lv=None):
""" Sample from predictive of PP for parameter lv.
Returns: list (len input_list) of np arrays (size (nsamp,1))."""
x_pred = np.stack(input_list)
if lv is None:
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' \
or self.modelp.model_str=='optmatern32':
lv = self.sample_list[0]
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampmatern32':
lv = self.sample_list[np.random.randint(len(self.sample_list))]
postmu, postcov = self.gp_post(self.data.X, self.data.y, x_pred, lv.ls,
lv.alpha, lv.sigma)
pred_list = list(sample_mvn(postmu, postcov, 1)) ###TODO: sample from this mean nsamp times
return self.get_reverse_transform(list(np.stack(pred_list).T), pred_list,
input_list)
def get_reverse_transform(self, pp1, pp2, input_list):
""" Apply reverse of data transform to ppred or pred """
pp1 = [pp1[i] + self.gp_mean_vec(input_list[i].reshape(1,-1)) for i in
range(len(input_list))]
pp2 = [psamp + self.gp_mean_vec(np.array(input_list)) for psamp in pp2]
return pp1, pp2
def gp_post(self, x_train, y_train, x_pred, ls, alpha, sigma, full_cov=True):
""" Compute parameters of GP posterior """
if self.modelp.model_str=='optmatern32' or \
self.modelp.model_str=='sampmatern32':
kernel = kern_matern32
else:
kernel = kern_exp_quad
k11_nonoise = kernel(x_train, x_train, ls, alpha)
lmat = get_cholesky_decomp(k11_nonoise, sigma, 'try_first')
smat = solve_upper_triangular(lmat.T, solve_lower_triangular(lmat, y_train))
k21 = kernel(x_pred, x_train, ls, alpha)
mu2 = k21.dot(smat)
k22 = kernel(x_pred, x_pred, ls, alpha)
vmat = solve_lower_triangular(lmat, k21.T)
k2 = k22 - vmat.T.dot(vmat)
if full_cov is False:
k2 = np.sqrt(np.diag(k2))
return mu2, k2
# Utilities
def print_str(self):
""" Print a description string """
print('*StanGpPP with modelp='+str(self.modelp)+'.')
print('-----')
================================================
FILE: bo/pp/pp_gp_stan_distmat.py
================================================
"""
Classes for GP models with Stan, using a given distance matrix.
"""
from argparse import Namespace
import time
import copy
import numpy as np
from scipy.spatial.distance import cdist
from bo.pp.pp_core import DiscPP
import bo.pp.stan.gp_distmat as gpstan
import bo.pp.stan.gp_distmat_fixedsig as gpstan_fixedsig
from bo.pp.gp.gp_utils import kern_exp_quad, kern_matern32, \
get_cholesky_decomp, solve_upper_triangular, solve_lower_triangular, \
sample_mvn, squared_euc_distmat, kern_distmat
from bo.util.print_utils import suppress_stdout_stderr
class StanGpDistmatPP(DiscPP):
""" Hierarchical GPs using a given distance matrix, implemented with Stan """
def __init__(self, data=None, modelp=None, printFlag=True):
""" Constructor """
self.set_model_params(modelp)
self.set_data(data)
self.ndimx = self.modelp.ndimx
self.set_model()
super(StanGpDistmatPP,self).__init__()
if printFlag:
self.print_str()
def set_model_params(self, modelp):
""" Set self.modelp """
if modelp is None:
pass #TODO
self.modelp = modelp
def set_data(self, data):
""" Set self.data """
if data is None:
pass #TODO
self.data_init = copy.deepcopy(data)
self.data = copy.deepcopy(self.data_init)
def set_model(self):
""" Set GP regression model """
self.model = self.get_model()
def get_model(self):
""" Returns GPRegression model """
if self.modelp.model_str=='optfixedsig' or \
self.modelp.model_str=='sampfixedsig':
return gpstan_fixedsig.get_model(print_status=True)
elif self.modelp.model_str=='opt' or self.modelp.model_str=='samp':
return gpstan.get_model(print_status=True)
elif self.modelp.model_str=='fixedparam':
return None
def infer_post_and_update_samples(self, seed=543210, print_result=False):
""" Update self.sample_list """
data_dict = self.get_stan_data_dict()
with suppress_stdout_stderr():
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt':
stanout = self.model.optimizing(data_dict, iter=self.modelp.infp.niter,
#seed=seed, as_vector=True, algorithm='Newton')
seed=seed, as_vector=True, algorithm='LBFGS')
elif self.modelp.model_str=='samp' or self.modelp.model_str=='sampfixedsig':
stanout = self.model.sampling(data_dict, iter=self.modelp.infp.niter +
self.modelp.infp.nwarmup, warmup=self.modelp.infp.nwarmup, chains=1,
seed=seed, refresh=1000)
elif self.modelp.model_str=='fixedparam':
stanout = None
print('-----')
self.sample_list = self.get_sample_list_from_stan_out(stanout)
if print_result: self.print_inference_result()
def get_stan_data_dict(self):
""" Return data dict for stan sampling method """
if self.modelp.model_str=='optfixedsig' or \
self.modelp.model_str=='sampfixedsig':
return {'ig1':self.modelp.kernp.ig1, 'ig2':self.modelp.kernp.ig2,
'n1':self.modelp.kernp.n1, 'n2':self.modelp.kernp.n2,
'sigma':self.modelp.kernp.sigma, 'D':self.ndimx,
'N':len(self.data.X), 'y':self.data.y.flatten(),
'distmat':self.get_distmat(self.data.X, self.data.X)}
elif self.modelp.model_str=='opt' or self.modelp.model_str=='samp':
return {'ig1':self.modelp.kernp.ig1, 'ig2':self.modelp.kernp.ig2,
'n1':self.modelp.kernp.n1, 'n2':self.modelp.kernp.n2,
'n3':self.modelp.kernp.n3, 'n4':self.modelp.kernp.n4,
'D':self.ndimx, 'N':len(self.data.X), 'y':self.data.y.flatten(),
'distmat':self.get_distmat(self.data.X, self.data.X)}
def get_distmat(self, xmat1, xmat2):
""" Get distance matrix """
# For now, will compute squared euc distance * .5, on self.data.X
return squared_euc_distmat(xmat1, xmat2, .5)
def get_sample_list_from_stan_out(self, stanout):
""" Convert stan output to sample_list """
if self.modelp.model_str=='optfixedsig':
return [Namespace(ls=stanout['rho'], alpha=stanout['alpha'],
sigma=self.modelp.kernp.sigma)]
elif self.modelp.model_str=='opt':
return [Namespace(ls=stanout['rho'], alpha=stanout['alpha'],
sigma=stanout['sigma'])]
elif self.modelp.model_str=='sampfixedsig':
sdict = stanout.extract(['rho','alpha'])
return [Namespace(ls=sdict['rho'][i], alpha=sdict['alpha'][i],
sigma=self.modelp.kernp.sigma) for i in range(sdict['rho'].shape[0])]
elif self.modelp.model_str=='samp':
sdict = stanout.extract(['rho','alpha','sigma'])
return [Namespace(ls=sdict['rho'][i], alpha=sdict['alpha'][i],
sigma=sdict['sigma'][i]) for i in range(sdict['rho'].shape[0])]
elif self.modelp.model_str=='fixedparam':
return [Namespace(ls=self.modelp.kernp.ls, alpha=self.modelp.kernp.alpha,
sigma=self.modelp.kernp.sigma)]
def print_inference_result(self):
""" Print results of stan inference """
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' or \
self.modelp.model_str=='fixedparam':
print('*ls pt est = '+str(self.sample_list[0].ls)+'.')
print('*alpha pt est = '+str(self.sample_list[0].alpha)+'.')
print('*sigma pt est = '+str(self.sample_list[0].sigma)+'.')
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampfixedsig':
ls_arr = np.array([ns.ls for ns in self.sample_list])
alpha_arr = np.array([ns.alpha for ns in self.sample_list])
sigma_arr = np.array([ns.sigma for ns in self.sample_list])
print('*ls mean = '+str(ls_arr.mean())+'.')
print('*ls std = '+str(ls_arr.std())+'.')
print('*alpha mean = '+str(alpha_arr.mean())+'.')
print('*alpha std = '+str(alpha_arr.std())+'.')
print('*sigma mean = '+str(sigma_arr.mean())+'.')
print('*sigma std = '+str(sigma_arr.std())+'.')
print('-----')
def sample_pp_post_pred(self, nsamp, input_list, full_cov=False, nloop=None):
""" Sample from posterior predictive of PP.
Inputs:
input_list - list of np arrays size=(-1,)
Returns:
list (len input_list) of np arrays (size=(nsamp,1))."""
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' or \
self.modelp.model_str=='fixedparam':
nloop = 1
sampids = [0]
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampfixedsig':
if nloop is None: nloop=nsamp
nsamp = int(nsamp/nloop)
sampids = np.random.randint(len(self.sample_list), size=(nloop,))
ppred_list = []
for i in range(nloop):
samp = self.sample_list[sampids[i]]
postmu, postcov = self.gp_post(self.data.X, self.data.y,
np.stack(input_list), samp.ls, samp.alpha, samp.sigma, full_cov)
if full_cov:
ppred_list.extend(list(sample_mvn(postmu, postcov, nsamp)))
else:
ppred_list.extend(list(np.random.normal(postmu.reshape(-1,),
postcov.reshape(-1,), size=(nsamp, len(input_list)))))
return list(np.stack(ppred_list).T), ppred_list
def sample_pp_pred(self, nsamp, input_list, lv=None):
""" Sample from predictive of PP for parameter lv.
Returns: list (len input_list) of np arrays (size (nsamp,1))."""
x_pred = np.stack(input_list)
if lv is None:
if self.modelp.model_str=='optfixedsig' or self.modelp.model_str=='opt' \
or self.modelp.model_str=='fixedparam':
lv = self.sample_list[0]
elif self.modelp.model_str=='samp' or \
self.modelp.model_str=='sampfixedsig':
lv = self.sample_list[np.random.randint(len(self.sample_list))]
postmu, postcov = self.gp_post(self.data.X, self.data.y, x_pred, lv.ls,
lv.alpha, lv.sigma)
pred_list = list(sample_mvn(postmu, postcov, 1)) ###TODO: sample from this mean nsamp times
return list(np.stack(pred_list).T), pred_list
def gp_post(self, x_train, y_train, x_pred, ls, alpha, sigma, full_cov=True):
""" Compute parameters of GP posterior """
kernel = lambda a, b, c, d: kern_distmat(a, b, c, d, self.get_distmat)
k11_nonoise = kernel(x_train, x_train, ls, alpha)
lmat = get_cholesky_decomp(k11_nonoise, sigma, 'try_first')
smat = solve_upper_triangular(lmat.T, solve_lower_triangular(lmat, y_train))
k21 = kernel(x_pred, x_train, ls, alpha)
mu2 = k21.dot(smat)
k22 = kernel(x_pred, x_pred, ls, alpha)
vmat = solve_lower_triangular(lmat, k21.T)
k2 = k22 - vmat.T.dot(vmat)
if full_cov is False:
k2 = np.sqrt(np.diag(k2))
return mu2, k2
# Utilities
def print_str(self):
""" Print a description string """
print('*StanGpDistmatPP with modelp='+str(self.modelp)+'.')
print('-----')
================================================
FILE: bo/pp/stan/__init__.py
================================================
"""
Code for defining and compiling models in Stan.
"""
================================================
FILE: bo/pp/stan/compile_stan.py
================================================
"""
Script to compile stan models
"""
#import pp_new.stan.gp_hier2 as gpstan
#import pp_new.stan.gp_hier3 as gpstan
#import pp_new.stan.gp_hier2_matern as gpstan
import pp_new.stan.gp_distmat as gpstan
#import pp_new.stan.gp_distmat_fixedsig as gpstan
# Recompile model and return it
model = gpstan.get_model(recompile=True)
================================================
FILE: bo/pp/stan/gp_distmat.py
================================================
"""
Functions to define and compile PPs in Stan, for model:
hierarchical GP (prior on rho, alpha, sigma) using a given distance matrix.
"""
import time
import pickle
import pystan
def get_model(recompile=False, print_status=True):
model_file_str = 'bo/pp/stan/hide_model/gp_distmat.pkl'
if recompile:
starttime = time.time()
model = pystan.StanModel(model_code=get_model_code())
buildtime = time.time()-starttime
with open(model_file_str,'wb') as f:
pickle.dump(model, f)
if print_status:
print('*Time taken to compile = '+ str(buildtime) +' seconds.\n-----')
print('*Model saved in file ' + model_file_str + '.\n-----')
else:
model = pickle.load(open(model_file_str,'rb'))
if print_status:
print('*Model loaded from file ' + model_file_str + '.\n-----')
return model
def get_model_code():
""" Parse modelp and return stan model code """
return """
data {
int N;
matrix[N, N] distmat;
vector[N] y;
real ig1;
real ig2;
real n1;
real n2;
real n3;
real n4;
}
parameters {
real rho;
real alpha;
real sigma;
}
model {
matrix[N, N] cov = square(alpha) * exp(-distmat / square(rho))
+ diag_matrix(rep_vector(square(sigma), N));
matrix[N, N] L_cov = cholesky_decompose(cov);
rho ~ inv_gamma(ig1, ig2);
alpha ~ normal(n1, n2);
sigma ~ normal(n3, n4);
y ~ multi_normal_cholesky(rep_vector(0, N), L_cov);
}
"""
if __name__ == '__main__':
get_model()
================================================
FILE: bo/pp/stan/gp_distmat_fixedsig.py
================================================
"""
Functions to define and compile PPs in Stan, for model:
hierarchical GP (prior on rho, alpha) and fixed sigma, using a given
distance matrix.
"""
import time
import pickle
import pystan
def get_model(recompile=False, print_status=True):
model_file_str = 'bo/pp/stan/hide_model/gp_distmat_fixedsig.pkl'
if recompile:
starttime = time.time()
model = pystan.StanModel(model_code=get_model_code())
buildtime = time.time()-starttime
with open(model_file_str,'wb') as f:
pickle.dump(model, f)
if print_status:
print('*Time taken to compile = '+ str(buildtime) +' seconds.\n-----')
print('*Model saved in file ' + model_file_str + '.\n-----')
else:
model = pickle.load(open(model_file_str,'rb'))
if print_status:
print('*Model loaded from file ' + model_file_str + '.\n-----')
return model
def get_model_code():
""" Parse modelp and return stan model code """
return """
data {
int N;
matrix[N, N] distmat;
vector[N] y;
real ig1;
real ig2;
real n1;
real n2;
real sigma;
}
parameters {
real rho;
real alpha;
}
model {
matrix[N, N] cov = square(alpha) * exp(-distmat / square(rho))
+ diag_matrix(rep_vector(square(sigma), N));
matrix[N, N] L_cov = cholesky_decompose(cov);
rho ~ inv_gamma(ig1, ig2);
alpha ~ normal(n1, n2);
y ~ multi_normal_cholesky(rep_vector(0, N), L_cov);
}
"""
if __name__ == '__main__':
get_model()
================================================
FILE: bo/pp/stan/gp_hier2.py
================================================
"""
Functions to define and compile PPs in Stan, for model:
hierarchical GP (prior on rho, alpha, sigma)
"""
import time
import pickle
import pystan
def get_model(recompile=False, print_status=True):
model_file_str = 'bo/pp/stan/hide_model/gp_hier2.pkl'
if recompile:
starttime = time.time()
model = pystan.StanModel(model_code=get_model_code())
buildtime = time.time()-starttime
with open(model_file_str,'wb') as f:
pickle.dump(model, f)
if print_status:
print('*Time taken to compile = '+ str(buildtime) +' seconds.\n-----')
print('*Model saved in file ' + model_file_str + '.\n-----')
else:
model = pickle.load(open(model_file_str,'rb'))
if print_status:
print('*Model loaded from file ' + model_file_str + '.\n-----')
return model
def get_model_code():
""" Parse modelp and return stan model code """
return """
data {
int D;
int N;
vector[D] x[N];
vector[N] y;
real ig1;
real ig2;
real n1;
real n2;
real n3;
real n4;
}
parameters {
real rho;
real alpha;
real sigma;
}
model {
matrix[N, N] cov = cov_exp_quad(x, alpha, rho)
+ diag_matrix(rep_vector(square(sigma), N));
matrix[N, N] L_cov = cholesky_decompose(cov);
rho ~ inv_gamma(ig1, ig2);
alpha ~ normal(n1, n2);
sigma ~ normal(n3, n4);
y ~ multi_normal_cholesky(rep_vector(0, N), L_cov);
}
"""
if __name__ == '__main__':
get_model()
================================================
FILE: bo/pp/stan/gp_hier2_matern.py
================================================
"""
Functions to define and compile PPs in Stan, for model: hierarchical GP (prior
on rho, alpha, sigma), with matern kernel
"""
import time
import pickle
import pystan
def get_model(recompile=False, print_status=True):
model_file_str = 'bo/pp/stan/hide_model/gp_hier2_matern.pkl'
if recompile:
starttime = time.time()
model = pystan.StanModel(model_code=get_model_code())
buildtime = time.time()-starttime
with open(model_file_str,'wb') as f:
pickle.dump(model, f)
if print_status:
print('*Time taken to compile = '+ str(buildtime) +' seconds.\n-----')
print('*Model saved in file ' + model_file_str + '.\n-----')
else:
model = pickle.load(open(model_file_str,'rb'))
if print_status:
print('*Model loaded from file ' + model_file_str + '.\n-----')
return model
def get_model_code():
""" Parse modelp and return stan model code """
return """
functions {
matrix distance_matrix_single(int N, vector[] x) {
matrix[N, N] distmat;
for(i in 1:(N-1)) {
for(j in (i+1):N) {
distmat[i, j] = distance(x[i], x[j]);
}
}
return distmat;
}
matrix matern_covariance(int N, matrix dist, real ls, real alpha_sq, int COVFN) {
matrix[N,N] S;
real dist_ls;
real sqrt3;
real sqrt5;
sqrt3=sqrt(3.0);
sqrt5=sqrt(5.0);
// exponential == Matern nu=1/2 , (p=0; nu=p+1/2)
if (COVFN==1) {
for(i in 1:(N-1)) {
for(j in (i+1):N) {
dist_ls = fabs(dist[i,j])/ls;
S[i,j] = alpha_sq * exp(- dist_ls );
}
}
}
// Matern nu= 3/2 covariance
else if (COVFN==2) {
for(i in 1:(N-1)) {
for(j in (i+1):N) {
dist_ls = fabs(dist[i,j])/ls;
S[i,j] = alpha_sq * (1 + sqrt3 * dist_ls) * exp(-sqrt3 * dist_ls);
}
}
}
// Matern nu=5/2 covariance
else if (COVFN==3) {
for(i in 1:(N-1)) {
for(j in (i+1):N) {
dist_ls = fabs(dist[i,j])/ls;
S[i,j] = alpha_sq * (1 + sqrt5 *dist_ls + 5* pow(dist_ls,2)/3) * exp(-sqrt5 *dist_ls);
}
}
}
// Matern as nu->Inf become Gaussian (aka squared exponential cov)
else if (COVFN==4) {
for(i in 1:(N-1)) {
for(j in (i+1):N) {
dist_ls = fabs(dist[i,j])/ls;
S[i,j] = alpha_sq * exp( -pow(dist_ls,2)/2 ) ;
}
}
}
// fill upper triangle
for(i in 1:(N-1)) {
for(j in (i+1):N) {
S[j,i] = S[i,j];
}
}
// create diagonal: nugget(nonspatial) + spatial variance + eps ensures positive definiteness
for(i in 1:N) {
S[i,i] = alpha_sq;
}
return S;
}
}
data {
int D;
int N;
vector[D] x[N];
vector[N] y;
real ig1;
real ig2;
real n1;
real n2;
real n3;
real n4;
int covid;
}
parameters {
real rho;
real alpha;
real sigma;
}
model {
matrix[N, N] distmat = distance_matrix_single(N, x);
matrix[N, N] cov = matern_covariance(N, distmat, rho, square(alpha), covid);
matrix[N, N] L_cov = cholesky_decompose(cov);
rho ~ inv_gamma(ig1, ig2);
alpha ~ normal(n1, n2);
sigma ~ normal(n3, n4);
y ~ multi_normal_cholesky(rep_vector(0, N), L_cov);
}
"""
if __name__ == '__main__':
get_model()
================================================
FILE: bo/pp/stan/gp_hier3.py
================================================
"""
Functions to define and compile PPs in Stan, for model:
hierarchical GP with uniform prior on rho, normal prior on alpha,
and fixed sigma
"""
import time
import pickle
import pystan
def get_model(recompile=False, print_status=True):
model_file_str = 'bo/pp/stan/hide_model/gp_hier3.pkl'
if recompile:
starttime = time.time()
model = pystan.StanModel(model_code=get_model_code())
buildtime = time.time()-starttime
with open(model_file_str,'wb') as f:
pickle.dump(model, f)
if print_status:
print('*Time taken to compile = '+ str(buildtime) +' seconds.\n-----')
print('*Model saved in file ' + model_file_str + '.\n-----')
else:
model = pickle.load(open(model_file_str,'rb'))
if print_status:
print('*Model loaded from file ' + model_file_str + '.\n-----')
return model
def get_model_code():
""" Parse modelp and return stan model code """
return """
data {
int D;
int N;
vector[D] x[N];
vector[N] y;
real u1;
real u2;
real n1;
real n2;
real sigma;
}
parameters {
real rho;
real alpha;
}
model {
matrix[N, N] cov = cov_exp_quad(x, alpha, rho)
+ diag_matrix(rep_vector(square(sigma), N));
matrix[N, N] L_cov = cholesky_decompose(cov);
rho ~ uniform(u1, u2);
alpha ~ normal(n1, n2);
y ~ multi_normal_cholesky(rep_vector(0, N), L_cov);
}
"""
if __name__ == '__main__':
get_model()
================================================
FILE: bo/util/__init__.py
================================================
"""
Miscellaneous utilities.
"""
================================================
FILE: bo/util/datatransform.py
================================================
"""
Classes for transforming data.
"""
from argparse import Namespace
import numpy as np
from sklearn.preprocessing import StandardScaler
#import sklearn.preprocessing as sklp
class DataTransformer(object):
""" Class for transforming data """
def __init__(self, datamat, printflag=True):
""" Constructor
Parameters:
datamat - numpy array (n x d) of data to be transformed
"""
self.datamat = datamat
self.set_transformers()
if printflag:
self.print_str()
def set_transformers(self):
""" Set transformers using self.datamat """
self.ss = StandardScaler()
self.ss.fit(self.datamat)
def transform_data(self, datamat=None):
""" Return transformed datamat (default self.datamat) """
if datamat is None:
datamat = self.datamat
return self.ss.transform(datamat)
def inv_transform_data(self, datamat):
""" Return inverse transform of datamat """
return self.ss.inverse_transform(datamat)
def print_str(self):
""" Print a description string """
print('*DataTransformer with self.datamat.shape = ' +
str(self.datamat.shape) + '.')
print('-----')
================================================
FILE: bo/util/print_utils.py
================================================
"""
Utilities for printing and output
"""
import os
class suppress_stdout_stderr(object):
''' A context manager for doing a "deep suppression" of stdout and stderr in
Python, i.e. will suppress all print, even if the print originates in a
compiled C/Fortran sub-function.
This will not suppress raised exceptions, since exceptions are printed
to stderr just before a script exits, and after the context manager has
exited (at least, I think that is why it lets exceptions through). '''
def __init__(self):
# Open a pair of null files
self.null_fds = [os.open(os.devnull, os.O_RDWR) for x in range(2)]
# Save the actual stdout (1) and stderr (2) file descriptors.
self.save_fds = [os.dup(1), os.dup(2)]
def __enter__(self):
# Assign the null pointers to stdout and stderr.
os.dup2(self.null_fds[0], 1)
os.dup2(self.null_fds[1], 2)
def __exit__(self, *_):
# Re-assign the real stdout/stderr back to (1) and (2)
os.dup2(self.save_fds[0], 1)
os.dup2(self.save_fds[1], 2)
# Close the null files
for fd in self.null_fds + self.save_fds:
os.close(fd)
================================================
FILE: darts/__init__.py
================================================
================================================
FILE: darts/arch.py
================================================
import numpy as np
import sys
import os
import copy
import random
sys.path.append(os.path.expanduser('~/darts/cnn'))
from train_class import Train
OPS = ['none',
'max_pool_3x3',
'avg_pool_3x3',
'skip_connect',
'sep_conv_3x3',
'sep_conv_5x5',
'dil_conv_3x3',
'dil_conv_5x5'
]
NUM_VERTICES = 4
INPUT_1 = 'c_k-2'
INPUT_2 = 'c_k-1'
class Arch:
def __init__(self, arch):
self.arch = arch
def serialize(self):
return self.arch
def query(self, epochs=50):
trainer = Train()
val_losses, test_losses = trainer.main(self.arch, epochs=epochs)
val_loss = 100 - np.mean(val_losses)
test_loss = 100 - test_losses[-1]
return val_loss, test_loss
@classmethod
def random_arch(cls):
# output a uniformly random architecture spec
# from the DARTS repository
# https://github.com/quark0/darts
normal = []
reduction = []
for i in range(NUM_VERTICES):
ops = np.random.choice(range(len(OPS)), NUM_VERTICES)
#input nodes for conv
nodes_in_normal = np.random.choice(range(i+2), 2, replace=False)
#input nodes for reduce
nodes_in_reduce = np.random.choice(range(i+2), 2, replace=False)
normal.extend([(nodes_in_normal[0], ops[0]), (nodes_in_normal[1], ops[1])])
reduction.extend([(nodes_in_reduce[0], ops[2]), (nodes_in_reduce[1], ops[3])])
return (normal, reduction)
def get_arch_list(self):
# convert tuple to list so that it is mutable
arch_list = []
for cell in self.arch:
arch_list.append([])
for pair in cell:
arch_list[-1].append([])
for num in pair:
arch_list[-1][-1].append(num)
return arch_list
def mutate(self, edits):
""" mutate a single arch """
# first convert tuple to array so that it is mutable
mutation = self.get_arch_list()
#make mutations
for _ in range(edits):
cell = np.random.choice(2)
pair = np.random.choice(len(OPS))
num = np.random.choice(2)
if num == 1:
mutation[cell][pair][num] = np.random.choice(len(OPS))
else:
inputs = pair // 2 + 2
choice = np.random.choice(inputs)
if pair % 2 == 0 and mutation[cell][pair+1][num] != choice:
mutation[cell][pair][num] = choice
elif pair % 2 != 0 and mutation[cell][pair-1][num] != choice:
mutation[cell][pair][num] = choice
return mutation
def get_paths(self):
""" return all paths from input to output """
path_builder = [[[], [], [], []], [[], [], [], []]]
paths = [[], []]
for i, cell in enumerate(self.arch):
for j in range(len(OPS)):
if cell[j][0] == 0:
path = [INPUT_1, OPS[cell[j][1]]]
path_builder[i][j//2].append(path)
paths[i].append(path)
elif cell[j][0] == 1:
path = [INPUT_2, OPS[cell[j][1]]]
path_builder[i][j//2].append(path)
paths[i].append(path)
else:
for path in path_builder[i][cell[j][0] - 2]:
path = [*path, OPS[cell[j][1]]]
path_builder[i][j//2].append(path)
paths[i].append(path)
# check if there are paths of length >=5
contains_long_path = [False, False]
if max([len(path) for path in paths[0]]) >= 5:
contains_long_path[0] = True
if max([len(path) for path in paths[1]]) >= 5:
contains_long_path[1] = True
return paths, contains_long_path
def get_path_indices(self, long_paths=True):
"""
compute the index of each path
There are 4 * (8^0 + ... + 8^4) paths total
If long_paths = False, we give a single boolean to all paths of
size 4, so there are only 4 * (1 + 8^0 + ... + 8^3) paths
"""
paths, contains_long_path = self.get_paths()
normal_paths, reduce_paths = paths
num_ops = len(OPS)
"""
Compute the max number of paths per input per cell.
Since there are two cells and two inputs per cell,
total paths = 4 * max_paths
"""
if not long_paths:
max_paths = 1 + sum([num_ops ** i for i in range(NUM_VERTICES)])
else:
max_paths = sum([num_ops ** i for i in range(NUM_VERTICES + 1)])
path_indices = []
# set the base index based on the cell and the input
for i, paths in enumerate((normal_paths, reduce_paths)):
for path in paths:
index = i * 2 * max_paths
if path[0] == INPUT_2:
index += max_paths
# recursively compute the index of the path
for j in range(NUM_VERTICES + 1):
if j == len(path) - 1:
path_indices.append(index)
break
elif j == (NUM_VERTICES - 1) and not long_paths:
path_indices.append(2 * (i + 1) * max_paths - 1)
break
else:
index += num_ops ** j * (OPS.index(path[j + 1]) + 1)
return (tuple(path_indices), contains_long_path)
def encode_paths(self, long_paths=True):
# output one-hot encoding of paths
path_indices, _ = self.get_path_indices(long_paths=long_paths)
num_ops = len(OPS)
if not long_paths:
max_paths = 1 + sum([num_ops ** i for i in range(NUM_VERTICES)])
else:
max_paths = sum([num_ops ** i for i in range(NUM_VERTICES + 1)])
path_encoding = np.zeros(4 * max_paths)
for index in path_indices:
path_encoding[index] = 1
return path_encoding
def path_distance(self, other):
# compute the distance between two architectures
# by comparing their path encodings
return np.sum(np.array(self.encode_paths() != np.array(other.encode_paths())))
================================================
FILE: data.py
================================================
import numpy as np
import pickle
import sys
import os
if 'search_space' not in os.environ or os.environ['search_space'] == 'nasbench':
from nasbench import api
from nas_bench.cell import Cell
elif os.environ['search_space'] == 'darts':
from darts.arch import Arch
elif os.environ['search_space'][:12] == 'nasbench_201':
from nas_201_api import NASBench201API as API
from nas_bench_201.cell import Cell
else:
print('Invalid search space environ in data.py')
sys.exit()
class Data:
def __init__(self,
search_space,
dataset='cifar10',
nasbench_folder='./',
loaded_nasbench=None):
self.search_space = search_space
self.dataset = dataset
if loaded_nasbench:
self.nasbench = loaded_nasbench
elif search_space == 'nasbench':
self.nasbench = api.NASBench(nasbench_folder + 'nasbench_only108.tfrecord')
elif search_space == 'nasbench_201':
self.nasbench = API(os.path.expanduser('~/nas-bench-201/NAS-Bench-201-v1_0-e61699.pth'))
elif search_space != 'darts':
print(search_space, 'is not a valid search space')
sys.exit()
def get_type(self):
return self.search_space
def query_arch(self,
arch=None,
train=True,
encoding_type='path',
cutoff=-1,
deterministic=True,
epochs=0):
arch_dict = {}
arch_dict['epochs'] = epochs
if self.search_space in ['nasbench', 'nasbench_201']:
if arch is None:
arch = Cell.random_cell(self.nasbench)
arch_dict['spec'] = arch
if encoding_type == 'adj':
encoding = Cell(**arch).encode_standard()
elif encoding_type == 'path':
encoding = Cell(**arch).encode_paths()
elif encoding_type == 'trunc_path':
encoding = Cell(**arch).encode_paths()[:cutoff]
else:
print('invalid encoding type')
arch_dict['encoding'] = encoding
if train:
arch_dict['val_loss'] = Cell(**arch).get_val_loss(self.nasbench,
deterministic=deterministic,
dataset=self.dataset)
arch_dict['test_loss'] = Cell(**arch).get_test_loss(self.nasbench,
dataset=self.dataset)
arch_dict['num_params'] = Cell(**arch).get_num_params(self.nasbench)
arch_dict['val_per_param'] = (arch_dict['val_loss'] - 4.8) * (arch_dict['num_params'] ** 0.5) / 100
else:
if arch is None:
arch = Arch.random_arch()
arch_dict['spec'] = arch
if encoding_type == 'path':
encoding = Arch(arch).encode_paths()
elif encoding_type == 'trunc_path':
encoding = Arch(arch).encode_paths()[:cutoff]
else:
encoding = arch
arch_dict['encoding'] = encoding
if train:
if epochs == 0:
epochs = 50
arch_dict['val_loss'], arch_dict['test_loss'] = Arch(arch).query(epochs=epochs)
return arch_dict
def mutate_arch(self,
arch,
mutation_rate=1.0):
if self.search_space in ['nasbench', 'nasbench_201']:
return Cell(**arch).mutate(self.nasbench,
mutation_rate=mutation_rate)
else:
return Arch(arch).mutate(int(mutation_rate))
def get_hash(self, arch):
# return the path indices of the architecture, used as a hash
if self.search_space == 'nasbench':
return Cell(**arch).get_path_indices()
elif self.search_space == 'darts':
return Arch(arch).get_path_indices()[0]
else:
return Cell(**arch).get_string()
def generate_random_dataset(self,
num=10,
train=True,
encoding_type='path',
cutoff=-1,
random='standard',
allow_isomorphisms=False,
deterministic_loss=True,
patience_factor=5):
"""
create a dataset of randomly sampled architectues
test for isomorphisms using a hash map of path indices
use patience_factor to avoid infinite loops
"""
data = []
dic = {}
tries_left = num * patience_factor
while len(data) < num:
tries_left -= 1
if tries_left <= 0:
break
arch_dict = self.query_arch(train=train,
encoding_type=encoding_type,
cutoff=cutoff,
deterministic=deterministic_loss)
h = self.get_hash(arch_dict['spec'])
if allow_isomorphisms or h not in dic:
dic[h] = 1
data.append(arch_dict)
return data
def get_candidates(self,
data,
num=100,
acq_opt_type='mutation',
encoding_type='path',
cutoff=-1,
loss='val_loss',
patience_factor=5,
deterministic_loss=True,
num_arches_to_mutate=1,
max_mutation_rate=1,
allow_isomorphisms=False):
"""
Creates a set of candidate architectures with mutated and/or random architectures
"""
candidates = []
# set up hash map
dic = {}
for d in data:
arch = d['spec']
h = self.get_hash(arch)
dic[h] = 1
if acq_opt_type in ['mutation', 'mutation_random']:
# mutate architectures with the lowest loss
best_arches = [arch['spec'] for arch in sorted(data, key=lambda i:i[loss])[:num_arches_to_mutate * patience_factor]]
# stop when candidates is size num
# use patience_factor instead of a while loop to avoid long or infinite runtime
for arch in best_arches:
if len(candidates) >= num:
break
for i in range(num // num_arches_to_mutate // max_mutation_rate):
for rate in range(1, max_mutation_rate + 1):
mutated = self.mutate_arch(arch, mutation_rate=rate)
arch_dict = self.query_arch(mutated,
train=False,
encoding_type=encoding_type,
cutoff=cutoff)
h = self.get_hash(mutated)
if allow_isomorphisms or h not in dic:
dic[h] = 1
candidates.append(arch_dict)
if acq_opt_type in ['random', 'mutation_random']:
# add randomly sampled architectures to the set of candidates
for _ in range(num * patience_factor):
if len(candidates) >= 2 * num:
break
arch_dict = self.query_arch(train=False,
encoding_type=encoding_type,
cutoff=cutoff)
h = self.get_hash(arch_dict['spec'])
if allow_isomorphisms or h not in dic:
dic[h] = 1
candidates.append(arch_dict)
return candidates
def remove_duplicates(self, candidates, data):
# input: two sets of architectues: candidates and data
# output: candidates with arches from data removed
dic = {}
for d in data:
dic[self.get_hash(d['spec'])] = 1
unduplicated = []
for candidate in candidates:
if self.get_hash(candidate['spec']) not in dic:
dic[self.get_hash(candidate['spec'])] = 1
unduplicated.append(candidate)
return unduplicated
def encode_data(self, dicts):
"""
method used by metann_runner.py (for Arch)
input: list of arch dictionary objects
output: xtrain (encoded architectures), ytrain (val loss)
"""
data = []
for dic in dicts:
arch = dic['spec']
encoding = Arch(arch).encode_paths()
data.append((arch, encoding, dic['val_loss_avg'], None))
return data
def get_arch_list(self,
aux_file_path,
iteridx=0,
num_top_arches=5,
max_edits=20,
num_repeats=5,
verbose=1):
# Method used for gp_bayesopt
if self.search_space == 'darts':
print('get_arch_list only supported for nasbench and nasbench_201')
sys.exit()
# load the list of architectures chosen by bayesopt so far
base_arch_list = pickle.load(open(aux_file_path, 'rb'))
top_arches = [archtuple[0] for archtuple in base_arch_list[:num_top_arches]]
if verbose:
top_5_loss = [archtuple[1][0] for archtuple in base_arch_list[:min(5, len(base_arch_list))]]
print('top 5 val losses {}'.format(top_5_loss))
# perturb the best k architectures
dic = {}
for archtuple in base_arch_list:
path_indices = Cell(**archtuple[0]).get_path_indices()
dic[path_indices] = 1
new_arch_list = []
for arch in top_arches:
for edits in range(1, max_edits):
for _ in range(num_repeats):
perturbation = Cell(**arch).perturb(self.nasbench, edits)
path_indices = Cell(**perturbation).get_path_indices()
if path_indices not in dic:
dic[path_indices] = 1
new_arch_list.append(perturbation)
# make sure new_arch_list is not empty
while len(new_arch_list) == 0:
for _ in range(100):
arch = Cell.random_cell(self.nasbench)
path_indices = Cell(**arch).get_path_indices()
if path_indices not in dic:
dic[path_indices] = 1
new_arch_list.append(arch)
return new_arch_list
@classmethod
def generate_distance_matrix(cls, arches_1, arches_2, distance):
# Method used for gp_bayesopt for nasbench
matrix = np.zeros([len(arches_1), len(arches_2)])
for i, arch_1 in enumerate(arches_1):
for j, arch_2 in enumerate(arches_2):
if distance == 'edit_distance':
matrix[i][j] = Cell(**arch_1).edit_distance(Cell(**arch_2))
elif distance == 'path_distance':
matrix[i][j] = Cell(**arch_1).path_distance(Cell(**arch_2))
elif distance == 'trunc_path_distance':
matrix[i][j] = Cell(**arch_1).path_distance(Cell(**arch_2))
elif distance == 'nasbot_distance':
matrix[i][j] = Cell(**arch_1).nasbot_distance(Cell(**arch_2))
else:
print('{} is an invalid distance'.format(distance))
sys.exit()
return matrix
================================================
FILE: meta_neural_net.py
================================================
import argparse
import itertools
import os
import random
import sys
import numpy as np
from matplotlib import pyplot as plt
from tensorflow import keras
import tensorflow as tf
from tensorflow.keras import backend as K
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
def mle_loss(y_true, y_pred):
# Minimum likelihood estimate loss function
mean = tf.slice(y_pred, [0, 0], [-1, 1])
var = tf.slice(y_pred, [0, 1], [-1, 1])
return 0.5 * tf.log(2*np.pi*var) + tf.square(y_true - mean) / (2*var)
def mape_loss(y_true, y_pred):
# Minimum absolute percentage error loss function
lower_bound = 4.5
fraction = tf.math.divide(tf.subtract(y_pred, lower_bound), \
tf.subtract(y_true, lower_bound))
return tf.abs(tf.subtract(fraction, 1))
class MetaNeuralnet:
def get_dense_model(self,
input_dims,
num_layers,
layer_width,
loss,
regularization):
input_layer = keras.layers.Input(input_dims)
model = keras.models.Sequential()
for _ in range(num_layers):
model.add(keras.layers.Dense(layer_width, activation='relu'))
model = model(input_layer)
if loss == 'mle':
mean = keras.layers.Dense(1)(model)
var = keras.layers.Dense(1)(model)
var = keras.layers.Activation(tf.math.softplus)(var)
output = keras.layers.concatenate([mean, var])
else:
if regularization == 0:
output = keras.layers.Dense(1)(model)
else:
reg = keras.regularizers.l1(regularization)
output = keras.layers.Dense(1, kernel_regularizer=reg)(model)
dense_net = keras.models.Model(inputs=input_layer, outputs=output)
return dense_net
def fit(self, xtrain, ytrain,
num_layers=10,
layer_width=20,
loss='mae',
epochs=200,
batch_size=32,
lr=.01,
verbose=0,
regularization=0,
**kwargs):
if loss == 'mle':
loss_fn = mle_loss
elif loss == 'mape':
loss_fn = mape_loss
else:
loss_fn = 'mae'
self.model = self.get_dense_model((xtrain.shape[1],),
loss=loss_fn,
num_layers=num_layers,
layer_width=layer_width,
regularization=regularization)
optimizer = keras.optimizers.Adam(lr=lr, beta_1=.9, beta_2=.99)
self.model.compile(optimizer=optimizer, loss=loss_fn)
#print(self.model.summary())
self.model.fit(xtrain, ytrain,
batch_size=batch_size,
epochs=epochs,
verbose=verbose)
train_pred = np.squeeze(self.model.predict(xtrain))
train_error = np.mean(abs(train_pred-ytrain))
return train_error
def predict(self, xtest):
return self.model.predict(xtest)
================================================
FILE: meta_neuralnet.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train a Meta Neural Network on NASBench\n",
"## Predict the accuracy of neural networks to within one percent!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"from nasbench import api\n",
"\n",
"from data import Data\n",
"from meta_neural_net import MetaNeuralnet"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# define a function to plot the meta neural networks\n",
"\n",
"def plot_meta_neuralnet(ytrain, train_pred, ytest, test_pred, max_disp=500, title=None):\n",
" \n",
" plt.scatter(ytrain[:max_disp], train_pred[:max_disp], label='training data', alpha=0.7, s=64)\n",
" plt.scatter(ytest[:max_disp], test_pred[:max_disp], label = 'test data', alpha=0.7, marker='^')\n",
"\n",
" # axis limits\n",
" plt.xlim((5, 15))\n",
" plt.ylim((5, 15))\n",
" ax_lim = np.array([np.min([plt.xlim()[0], plt.ylim()[0]]),\n",
" np.max([plt.xlim()[1], plt.ylim()[1]])])\n",
" plt.xlim(ax_lim)\n",
" plt.ylim(ax_lim)\n",
" \n",
" # 45-degree line\n",
" plt.plot(ax_lim, ax_lim, 'k:') \n",
" \n",
" plt.gca().set_aspect('equal', adjustable='box')\n",
" plt.title(title)\n",
" plt.legend(loc='best')\n",
" plt.xlabel('true percent error')\n",
" plt.ylabel('predicted percent error')\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# load the NASBench dataset\n",
"# takes about 1 minute to load the nasbench dataset\n",
"search_space = Data('nasbench')\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# method which runs a meta neural network experiment\n",
"def meta_neuralnet_experiment(params, \n",
" ns=[100, 500], \n",
" num_ensemble=3, \n",
" test_size=500,\n",
" cutoff=40,\n",
" plot=True):\n",
" \n",
" for n in ns:\n",
" for encoding_type in ['adj', 'path']:\n",
"\n",
" train_data = search_space.generate_random_dataset(num=n, \n",
" encoding_type=encoding_type,\n",
" cutoff=cutoff)\n",
" \n",
" test_data = search_space.generate_random_dataset(num=test_size, \n",
" encoding_type=encoding_type,\n",
" cutoff=cutoff)\n",
" \n",
" print(len(test_data))\n",
" test_data = search_space.remove_duplicates(test_data, train_data)\n",
" print(len(test_data))\n",
" \n",
" xtrain = np.array([d['encoding'] for d in train_data])\n",
" ytrain = np.array([d['val_loss'] for d in train_data])\n",
"\n",
" xtest = np.array([d['encoding'] for d in test_data])\n",
" ytest = np.array([d['val_loss'] for d in test_data])\n",
"\n",
" train_errors = []\n",
" test_errors = []\n",
" meta_neuralnet = MetaNeuralnet()\n",
" for _ in range(num_ensemble): \n",
" meta_neuralnet.fit(xtrain, ytrain, **params)\n",
" train_pred = np.squeeze(meta_neuralnet.predict(xtrain))\n",
" train_error = np.mean(abs(train_pred-ytrain))\n",
" train_errors.append(train_error)\n",
" test_pred = np.squeeze(meta_neuralnet.predict(xtest)) \n",
" test_error = np.mean(abs(test_pred-ytest))\n",
" test_errors.append(test_error)\n",
"\n",
" train_error = np.round(np.mean(train_errors, axis=0), 3)\n",
" test_error = np.round(np.mean(test_errors, axis=0), 3)\n",
" print('Meta neuralnet training size: {}, encoding type: {}'.format(n, encoding_type))\n",
" print('Train error: {}, test error: {}'.format(train_error, test_error))\n",
"\n",
" if plot:\n",
" if encoding_type == 'path':\n",
" title = 'Path encoding, training set size {}'.format(n)\n",
" else:\n",
" title = 'Adjacency list encoding, training set size {}'.format(n) \n",
"\n",
" plot_meta_neuralnet(ytrain, train_pred, ytest, test_pred, title=title)\n",
" plt.show() \n",
" print('correlation', np.corrcoef(ytest, test_pred)[1,0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"meta_neuralnet_params = {'loss':'mae', 'num_layers':10, 'layer_width':20, 'epochs':200, \\\n",
" 'batch_size':32, 'lr':.01, 'regularization':0, 'verbose':0}\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"meta_neuralnet_experiment(meta_neuralnet_params)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: metann_runner.py
================================================
import argparse
import time
import logging
import sys
import os
import pickle
import numpy as np
from acquisition_functions import acq_fn
from data import Data
from meta_neural_net import MetaNeuralnet
"""
meta neural net runner is used in run_experiments_parallel
- loads data by opening k*i pickle files from previous iterations
- trains a meta neural network and predicts accuracy of all candidates
- outputs k pickle files of the architecture to be trained next
"""
def run_meta_neuralnet(search_space, dicts,
k=10,
verbose=1,
num_ensemble=5,
epochs=10000,
lr=0.00001,
loss='scaled',
explore_type='its',
explore_factor=0.5):
# data: list of arch dictionary objects
# trains a meta neural network
# returns list of k arch dictionary objects - the k best predicted
results = []
meta_neuralnet = MetaNeuralnet()
data = search_space.encode_data(dicts)
xtrain = np.array([d[1] for d in data])
ytrain = np.array([d[2] for d in data])
candidates = search_space.get_candidates(data,
acq_opt_type='mutation_random',
encode_paths=True,
allow_isomorphisms=True,
deterministic_loss=None)
xcandidates = np.array([c[1] for c in candidates])
candidates_specs = [c[0] for c in candidates]
predictions = []
# train an ensemble of neural networks
train_error = 0
for _ in range(num_ensemble):
meta_neuralnet = MetaNeuralnet()
train_error += meta_neuralnet.fit(xtrain, ytrain,
loss=loss,
epochs=epochs,
lr=lr)
predictions.append(np.squeeze(meta_neuralnet.predict(xcandidates)))
train_error /= num_ensemble
if verbose:
print('Meta neural net train error: {}'.format(train_error))
sorted_indices = acq_fn(predictions, explore_type)
top_k_candidates = [candidates_specs[i] for i in sorted_indices[:k]]
candidates_dict = []
for candidate in top_k_candidates:
d = {}
d['spec'] = candidate
candidates_dict.append(d)
return candidates_dict
def run(args):
save_dir = '{}/'.format(args.experiment_name)
if not os.path.exists(save_dir):
os.mkdir(save_dir)
query = args.query
k = args.k
trained_prefix = args.trained_filename
untrained_prefix = args.untrained_filename
threshold = args.threshold
search_space = Data('darts')
# if it's the first iteration, choose k arches at random to train
if query == 0:
print('about to generate {} random'.format(k))
data = search_space.generate_random_dataset(num=k, train=False)
arches = [d['spec'] for d in data]
next_arches = []
for arch in arches:
d = {}
d['spec'] = arch
next_arches.append(d)
else:
# get the data from prior iterations from pickle files
data = []
for i in range(query):
filepath = '{}{}_{}.pkl'.format(save_dir, trained_prefix, i)
with open(filepath, 'rb') as f:
arch = pickle.load(f)
data.append(arch)
print('Iteration {}'.format(query))
print('Data from last round')
print(data)
# run the meta neural net to output the next arches
next_arches = run_meta_neuralnet(search_space, data, k=k)
print('next batch')
print(next_arches)
# output the new arches to pickle files
for i in range(k):
index = query + i
filepath = '{}{}_{}.pkl'.format(save_dir, untrained_prefix, index)
next_arches[i]['index'] = index
next_arches[i]['filepath'] = filepath
with open(filepath, 'wb') as f:
pickle.dump(next_arches[i], f)
def main(args):
#set up save dir
save_dir = './'
#set up logging
log_format = '%(asctime)s %(message)s'
logging.basicConfig(stream=sys.stdout, level=logging.INFO,
format=log_format, datefmt='%m/%d %I:%M:%S %p')
fh = logging.FileHandler(os.path.join(save_dir, 'log.txt'))
fh.setFormatter(logging.Formatter(log_format))
logging.getLogger().addHandler(fh)
logging.info(args)
run(args)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Args for meta neural net')
parser.add_argument('--experiment_name', type=str, default='darts_test', help='Folder for input/output files')
parser.add_argument('--params', type=str, default='test', help='Which set of params to use')
parser.add_argument('--query', type=int, default=0, help='Which query is Neural BayesOpt on')
parser.add_argument('--trained_filename', type=str, default='trained_spec', help='name of input files')
parser.add_argument('--untrained_filename', type=str, default='untrained_spec', help='name of output files')
parser.add_argument('--k', type=int, default=10, help='number of arches to train per iteration')
parser.add_argument('--threshold', type=int, default=20, help='throw out arches with val loss above threshold')
args = parser.parse_args()
main(args)
================================================
FILE: nas_algorithms.py
================================================
import itertools
import os
import pickle
import sys
import copy
import numpy as np
import tensorflow as tf
from argparse import Namespace
from data import Data
def run_nas_algorithm(algo_params, search_space, mp):
# run nas algorithm
ps = copy.deepcopy(algo_params)
algo_name = ps.pop('algo_name')
if algo_name == 'random':
data = random_search(search_space, **ps)
elif algo_name == 'evolution':
data = evolution_search(search_space, **ps)
elif algo_name == 'bananas':
data = bananas(search_space, mp, **ps)
elif algo_name == 'gp_bayesopt':
data = gp_bayesopt_search(search_space, **ps)
elif algo_name == 'dngo':
data = dngo_search(search_space, **ps)
else:
print('invalid algorithm name')
sys.exit()
k = 10
if 'k' in ps:
k = ps['k']
total_queries = 150
if 'total_queries' in ps:
total_queries = ps['total_queries']
loss = 'val_loss'
if 'loss' in ps:
loss = ps['loss']
return compute_best_test_losses(data, k, total_queries, loss), data
def compute_best_test_losses(data, k, total_queries, loss):
"""
Given full data from a completed nas algorithm,
output the test error of the arch with the best val error
after every multiple of k
"""
results = []
for query in range(k, total_queries + k, k):
best_arch = sorted(data[:query], key=lambda i:i[loss])[0]
test_error = best_arch['test_loss']
results.append((query, test_error))
return results
def random_search(search_space,
total_queries=150,
loss='val_loss',
deterministic=True,
verbose=1):
"""
random search
"""
data = search_space.generate_random_dataset(num=total_queries,
encoding_type='adj',
deterministic_loss=deterministic)
if verbose:
top_5_loss = sorted([d[loss] for d in data])[:min(5, len(data))]
print('random, query {}, top 5 losses {}'.format(total_queries, top_5_loss))
return data
def evolution_search(search_space,
total_queries=150,
num_init=10,
k=10,
loss='val_loss',
population_size=30,
tournament_size=10,
mutation_rate=1.0,
deterministic=True,
regularize=True,
verbose=1):
"""
regularized evolution
"""
data = search_space.generate_random_dataset(num=num_init,
deterministic_loss=deterministic)
losses = [d[loss] for d in data]
query = num_init
population = [i for i in range(min(num_init, population_size))]
while query <= total_queries:
# evolve the population by mutating the best architecture
# from a random subset of the population
sample = np.random.choice(population, tournament_size)
best_index = sorted([(i, losses[i]) for i in sample], key=lambda i:i[1])[0][0]
mutated = search_space.mutate_arch(data[best_index]['spec'],
mutation_rate=mutation_rate)
arch_dict = search_space.query_arch(mutated, deterministic=deterministic)
data.append(arch_dict)
losses.append(arch_dict[loss])
population.append(len(data) - 1)
# kill the oldest (or worst) from the population
if len(population) >= population_size:
if regularize:
oldest_index = sorted([i for i in population])[0]
population.remove(oldest_index)
else:
worst_index = sorted([(i, losses[i]) for i in population], key=lambda i:i[1])[-1][0]
population.remove(worst_index)
if verbose and (query % k == 0):
top_5_loss = sorted([d[loss] for d in data])[:min(5, len(data))]
print('evolution, query {}, top 5 losses {}'.format(query, top_5_loss))
query += 1
return data
def bananas(search_space,
metann_params,
num_init=10,
k=10,
loss='val_loss',
total_queries=150,
num_ensemble=5,
acq_opt_type='mutation',
num_arches_to_mutate=1,
explore_type='its',
encoding_type='trunc_path',
cutoff=40,
deterministic=True,
verbose=1):
"""
Bayesian optimization with a neural network model
"""
from acquisition_functions import acq_fn
from meta_neural_net import MetaNeuralnet
data = search_space.generate_random_dataset(num=num_init,
encoding_type=encoding_type,
cutoff=cutoff,
deterministic_loss=deterministic)
query = num_init + k
while query <= total_queries:
xtrain = np.array([d['encoding'] for d in data])
ytrain = np.array([d[loss] for d in data])
if (query == num_init + k) and verbose:
print('bananas xtrain shape', xtrain.shape)
print('bananas ytrain shape', ytrain.shape)
# get a set of candidate architectures
candidates = search_space.get_candidates(data,
acq_opt_type=acq_opt_type,
encoding_type=encoding_type,
cutoff=cutoff,
num_arches_to_mutate=num_arches_to_mutate,
loss=loss,
deterministic_loss=deterministic)
xcandidates = np.array([c['encoding'] for c in candidates])
candidate_predictions = []
# train an ensemble of neural networks
train_error = 0
for _ in range(num_ensemble):
meta_neuralnet = MetaNeuralnet()
train_error += meta_neuralnet.fit(xtrain, ytrain, **metann_params)
# predict the validation loss of the candidate architectures
candidate_predictions.append(np.squeeze(meta_neuralnet.predict(xcandidates)))
# clear the tensorflow graph
tf.reset_default_graph()
tf.keras.backend.clear_session()
train_error /= num_ensemble
if verbose:
print('query {}, Meta neural net train error: {}'.format(query, train_error))
# compute the acquisition function for all the candidate architectures
candidate_indices = acq_fn(candidate_predictions, explore_type)
# add the k arches with the minimum acquisition function values
for i in candidate_indices[:k]:
arch_dict = search_space.query_arch(candidates[i]['spec'],
encoding_type=encoding_type,
cutoff=cutoff,
deterministic=deterministic)
data.append(arch_dict)
if verbose:
top_5_loss = sorted([(d[loss], d['epochs']) for d in data], key=lambda d: d[0])[:min(5, len(data))]
print('bananas, query {}, top 5 losses (loss, test, epoch): {}'.format(query, top_5_loss))
recent_10_loss = [(d[loss], d['epochs']) for d in data[-10:]]
print('bananas, query {}, most recent 10 (loss, test, epoch): {}'.format(query, recent_10_loss))
query += k
return data
def gp_bayesopt_search(search_space,
num_init=10,
k=10,
total_queries=150,
distance='edit_distance',
deterministic=True,
tmpdir='./temp',
max_iter=200,
mode='single_process',
nppred=1000):
"""
Bayesian optimization with a GP prior
"""
from bo.bo.probo import ProBO
# set up the path for auxiliary pickle files
if not os.path.exists(tmpdir):
os.mkdir(tmpdir)
aux_file_path = os.path.join(tmpdir, 'aux.pkl')
num_iterations = total_queries - num_init
# black-box function that bayesopt will optimize
def fn(arch):
return search_space.query_arch(arch, deterministic=deterministic)['val_loss']
# set all the parameters for the various BayesOpt classes
fhp = Namespace(fhstr='object', namestr='train')
domp = Namespace(dom_str='list', set_domain_list_auto=True,
aux_file_path=aux_file_path,
distance=distance)
modelp = Namespace(kernp=Namespace(ls=3., alpha=1.5, sigma=1e-5),
infp=Namespace(niter=num_iterations, nwarmup=500),
distance=distance, search_space=search_space.get_type())
amp = Namespace(am_str='mygpdistmat_ucb', nppred=nppred, modelp=modelp)
optp = Namespace(opt_str='rand', max_iter=max_iter)
makerp = Namespace(domp=domp, amp=amp, optp=optp)
probop = Namespace(niter=num_iterations, fhp=fhp,
makerp=makerp, tmpdir=tmpdir, mode=mode)
data = Namespace()
# Set up initial data
init_data = search_space.generate_random_dataset(num=num_init,
deterministic_loss=deterministic)
data.X = [d['spec'] for d in init_data]
data.y = np.array([[d['val_loss']] for d in init_data])
# initialize aux file
pairs = [(data.X[i], data.y[i]) for i in range(len(data.y))]
pairs.sort(key=lambda x: x[1])
with open(aux_file_path, 'wb') as f:
pickle.dump(pairs, f)
# run Bayesian Optimization
bo = ProBO(fn, search_space, aux_file_path, data, probop, True)
bo.run_bo()
# get the validation and test loss for all architectures chosen by BayesOpt
results = []
for arch in data.X:
archtuple = search_space.query_arch(arch)
results.append(archtuple)
return results
def dngo_search(search_space,
num_init=10,
k=10,
loss='val_loss',
total_queries=150,
encoding_type='path',
cutoff=40,
acq_opt_type='mutation',
explore_type='ucb',
deterministic=True,
verbose=True):
import torch
from pybnn import DNGO
from pybnn.util.normalization import zero_mean_unit_var_normalization, zero_mean_unit_var_denormalization
from acquisition_functions import acq_fn
def fn(arch):
return search_space.query_arch(arch, deterministic=deterministic)[loss]
# set up initial data
data = search_space.generate_random_dataset(num=num_init,
encoding_type=encoding_type,
cutoff=cutoff,
deterministic_loss=deterministic)
query = num_init + k
while query <= total_queries:
# set up data
x = np.array([d['encoding'] for d in data])
y = np.array([d[loss] for d in data])
# get a set of candidate architectures
candidates = search_space.get_candidates(data,
acq_opt_type=acq_opt_type,
encoding_type=encoding_type,
cutoff=cutoff,
deterministic_loss=deterministic)
xcandidates = np.array([d['encoding'] for d in candidates])
# train the model
model = DNGO(do_mcmc=False)
model.train(x, y, do_optimize=True)
predictions = model.predict(xcandidates)
candidate_indices = acq_fn(np.array(predictions), explore_type)
# add the k arches with the minimum acquisition function values
for i in candidate_indices[:k]:
arch_dict = search_space.query_arch(candidates[i]['spec'],
encoding_type=encoding_type,
cutoff=cutoff,
deterministic=deterministic)
data.append(arch_dict)
if verbose:
top_5_loss = sorted([(d[loss], d['epochs']) for d in data], key=lambda d: d[0])[:min(5, len(data))]
print('dngo, query {}, top 5 val losses (val, test, epoch): {}'.format(query, top_5_loss))
recent_10_loss = [(d[loss], d['epochs']) for d in data[-10:]]
print('dngo, query {}, most recent 10 (val, test, epoch): {}'.format(query, recent_10_loss))
query += k
return data
================================================
FILE: nas_bench/__init__.py
================================================
================================================
FILE: nas_bench/cell.py
================================================
import numpy as np
import copy
import itertools
import random
import sys
import os
import pickle
from nasbench import api
INPUT = 'input'
OUTPUT = 'output'
CONV3X3 = 'conv3x3-bn-relu'
CONV1X1 = 'conv1x1-bn-relu'
MAXPOOL3X3 = 'maxpool3x3'
OPS = [CONV3X3, CONV1X1, MAXPOOL3X3]
NUM_VERTICES = 7
OP_SPOTS = NUM_VERTICES - 2
MAX_EDGES = 9
class Cell:
def __init__(self, matrix, ops):
self.matrix = matrix
self.ops = ops
def serialize(self):
return {
'matrix': self.matrix,
'ops': self.ops
}
def modelspec(self):
return api.ModelSpec(matrix=self.matrix, ops=self.ops)
@classmethod
def random_cell(cls, nasbench):
"""
From the NASBench repository
one-hot adjacency matrix
draw [0,1] for each slot in the adjacency matrix
"""
while True:
matrix = np.random.choice(
[0, 1], size=(NUM_VERTICES, NUM_VERTICES))
matrix = np.triu(matrix, 1)
ops = np.random.choice(OPS, size=NUM_VERTICES).tolist()
ops[0] = INPUT
ops[-1] = OUTPUT
spec = api.ModelSpec(matrix=matrix, ops=ops)
if nasbench.is_valid(spec):
return {
'matrix': matrix,
'ops': ops
}
def get_val_loss(self, nasbench, deterministic=1, patience=50, epochs=None, dataset=None):
if not deterministic:
# output one of the three validation accuracies at random
if epochs:
return (100*(1 - nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops), epochs=epochs)['validation_accuracy']))
else:
return (100*(1 - nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops))['validation_accuracy']))
else:
# query the api until we see all three accuracies, then average them
# a few architectures only have two accuracies, so we use patience to avoid an infinite loop
accs = []
while len(accs) < 3 and patience > 0:
patience -= 1
if epochs:
acc = nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops), epochs=epochs)['validation_accuracy']
else:
acc = nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops))['validation_accuracy']
if acc not in accs:
accs.append(acc)
return round(100*(1-np.mean(accs)), 4)
def get_test_loss(self, nasbench, patience=50, epochs=None, dataset=None):
"""
query the api until we see all three accuracies, then average them
a few architectures only have two accuracies, so we use patience to avoid an infinite loop
"""
accs = []
while len(accs) < 3 and patience > 0:
patience -= 1
if epochs:
acc = nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops), epochs=epochs)['test_accuracy']
else:
acc = nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops))['test_accuracy']
if acc not in accs:
accs.append(acc)
return round(100*(1-np.mean(accs)), 4)
def get_num_params(self, nasbench):
return nasbench.query(api.ModelSpec(matrix=self.matrix, ops=self.ops))['trainable_parameters']
def perturb(self, nasbench, edits=1):
"""
create new perturbed cell
inspird by https://github.com/google-research/nasbench
"""
new_matrix = copy.deepcopy(self.matrix)
new_ops = copy.deepcopy(self.ops)
for _ in range(edits):
while True:
if np.random.random() < 0.5:
for src in range(0, NUM_VERTICES - 1):
for dst in range(src+1, NUM_VERTICES):
new_matrix[src][dst] = 1 - new_matrix[src][dst]
else:
for ind in range(1, NUM_VERTICES - 1):
available = [op for op in OPS if op != new_ops[ind]]
new_ops[ind] = np.random.choice(available)
new_spec = api.ModelSpec(new_matrix, new_ops)
if nasbench.is_valid(new_spec):
break
return {
'matrix': new_matrix,
'ops': new_ops
}
def mutate(self,
nasbench,
mutation_rate=1.0,
patience=5000):
"""
A stochastic approach to perturbing the cell
inspird by https://github.com/google-research/nasbench
"""
p = 0
while p < patience:
p += 1
new_matrix = copy.deepcopy(self.matrix)
new_ops = copy.deepcopy(self.ops)
edge_mutation_prob = mutation_rate / (NUM_VERTICES * (NUM_VERTICES - 1) / 2)
# flip each edge w.p. so expected flips is 1. same for ops
for src in range(0, NUM_VERTICES - 1):
for dst in range(src + 1, NUM_VERTICES):
if random.random() < edge_mutation_prob:
new_matrix[src, dst] = 1 - new_matrix[src, dst]
op_mutation_prob = mutation_rate / OP_SPOTS
for ind in range(1, OP_SPOTS + 1):
if random.random() < op_mutation_prob:
available = [o for o in OPS if o != new_ops[ind]]
new_ops[ind] = random.choice(available)
new_spec = api.ModelSpec(new_matrix, new_ops)
if nasbench.is_valid(new_spec):
return {
'matrix': new_matrix,
'ops': new_ops
}
return self.mutate(nasbench, mutation_rate+1)
def encode_standard(self):
"""
compute the "standard" encoding,
i.e. adjacency matrix + op list encoding
"""
encoding_length = (NUM_VERTICES ** 2 - NUM_VERTICES) // 2 + OP_SPOTS
encoding = np.zeros((encoding_length))
dic = {CONV1X1: 0., CONV3X3: 0.5, MAXPOOL3X3: 1.0}
n = 0
for i in range(NUM_VERTICES - 1):
for j in range(i+1, NUM_VERTICES):
encoding[n] = self.matrix[i][j]
n += 1
for i in range(1, NUM_VERTICES - 1):
encoding[-i] = dic[self.ops[i]]
return tuple(encoding)
def get_paths(self):
"""
return all paths from input to output
"""
paths = []
for j in range(0, NUM_VERTICES):
paths.append([[]]) if self.matrix[0][j] else paths.append([])
# create paths sequentially
for i in range(1, NUM_VERTICES - 1):
for j in range(1, NUM_VERTICES):
if self.matrix[i][j]:
for path in paths[i]:
paths[j].append([*path, self.ops[i]])
return paths[-1]
def get_path_indices(self):
"""
compute the index of each path
There are 3^0 + ... + 3^5 paths total.
(Paths can be length 0 to 5, and for each path, for each node, there
are three choices for the operation.)
"""
paths = self.get_paths()
mapping = {CONV3X3: 0, CONV1X1: 1, MAXPOOL3X3: 2}
path_indices = []
for path in paths:
index = 0
for i in range(NUM_VERTICES - 1):
if i == len(path):
path_indices.append(index)
break
else:
index += len(OPS) ** i * (mapping[path[i]] + 1)
path_indices.sort()
return tuple(path_indices)
def encode_paths(self):
""" output one-hot encoding of paths """
num_paths = sum([len(OPS) ** i for i in range(OP_SPOTS + 1)])
path_indices = self.get_path_indices()
encoding = np.zeros(num_paths)
for index in path_indices:
encoding[index] = 1
return encoding
def path_distance(self, other):
"""
compute the distance between two architectures
by comparing their path encodings
"""
return np.sum(np.array(self.encode_paths() != np.array(other.encode_paths())))
def trunc_path_distance(self, other, cutoff=40):
"""
compute the distance between two architectures
by comparing their path encodings
"""
encoding = self.encode_paths()[:cutoff]
other_encoding = other.encode_paths()[:cutoff]
return np.sum(np.array(encoding) != np.array(other_encoding))
def edit_distance(self, other):
"""
compute the distance between two architectures
by comparing their adjacency matrices and op lists
"""
graph_dist = np.sum(np.array(self.matrix) != np.array(other.matrix))
ops_dist = np.sum(np.array(self.ops) != np.array(other.ops))
return (graph_dist + ops_dist)
def nasbot_distance(self, other):
# distance based on optimal transport between row sums, column sums, and ops
row_sums = sorted(np.array(self.matrix).sum(axis=0))
col_sums = sorted(np.array(self.matrix).sum(axis=1))
other_row_sums = sorted(np.array(other.matrix).sum(axis=0))
other_col_sums = sorted(np.array(other.matrix).sum(axis=1))
row_dist = np.sum(np.abs(np.subtract(row_sums, other_row_sums)))
col_dist = np.sum(np.abs(np.subtract(col_sums, other_col_sums)))
counts = [self.ops.count(op) for op in OPS]
other_counts = [other.ops.count(op) for op in OPS]
ops_dist = np.sum(np.abs(np.subtract(counts, other_counts)))
return (row_dist + col_dist + ops_dist)
================================================
FILE: nas_bench_201/__init__.py
================================================
================================================
FILE: nas_bench_201/cell.py
================================================
import numpy as np
import copy
import itertools
import random
import sys
import os
import pickle
OPS = ['avg_pool_3x3', 'nor_conv_1x1', 'nor_conv_3x3', 'none', 'skip_connect']
NUM_OPS = len(OPS)
OP_SPOTS = 6
LONGEST_PATH_LENGTH = 3
class Cell:
def __init__(self, string):
self.string = string
def get_string(self):
return self.string
def serialize(self):
return {
'string':self.string
}
@classmethod
def random_cell(cls, nasbench, max_nodes=4):
"""
From the AutoDL-Projects repository
"""
ops = []
for i in range(OP_SPOTS):
op = random.choice(OPS)
ops.append(op)
return {'string':cls.get_string_from_ops(ops)}
def get_runtime(self, nasbench, dataset='cifar100'):
return nasbench.query_by_index(index, dataset).get_eval('x-valid')['time']
def get_val_loss(self, nasbench, deterministic=1, dataset='cifar100'):
index = nasbench.query_index_by_arch(self.string)
if dataset == 'cifar10':
results = nasbench.query_by_index(index, 'cifar10-valid')
else:
results = nasbench.query_by_index(index, dataset)
accs = []
for key in results.keys():
accs.append(results[key].get_eval('x-valid')['accuracy'])
if deterministic:
return round(100-np.mean(accs), 10)
else:
return round(100-np.random.choice(accs), 10)
def get_test_loss(self, nasbench, dataset='cifar100', deterministic=1):
index = nasbench.query_index_by_arch(self.string)
results = nasbench.query_by_index(index, dataset)
accs = []
for key in results.keys():
accs.append(results[key].get_eval('ori-test')['accuracy'])
if deterministic:
return round(100-np.mean(accs), 4)
else:
return round(100-np.random.choice(accs), 4)
def get_op_list(self):
# given a string, get the list of operations
tokens = self.string.split('|')
ops = [t.split('~')[0] for i,t in enumerate(tokens) if i not in [0,2,5,9]]
return ops
def get_num(self):
# compute the unique number of the architecture, in [0, 15624]
ops = self.get_op_list()
index = 0
for i, op in enumerate(ops):
index += OPS.index(op) * NUM_OPS ** i
return index
@classmethod
def get_string_from_ops(cls, ops):
# given a list of operations, get the string
strings = ['|']
nodes = [0, 0, 1, 0, 1, 2]
for i, op in enumerate(ops):
strings.append(op+'~{}|'.format(nodes[i]))
if i < len(nodes) - 1 and nodes[i+1] == 0:
strings.append('+|')
return ''.join(strings)
def perturb(self, nasbench,
mutation_rate=1):
# more deterministic version of mutate
ops = self.get_op_list()
new_ops = []
num = np.random.choice(len(ops))
for i, op in enumerate(ops):
if i == num:
available = [o for o in OPS if o != op]
new_ops.append(np.random.choice(available))
else:
new_ops.append(op)
return {'string':self.get_string_from_ops(new_ops)}
def mutate(self,
nasbench,
mutation_rate=1.0,
patience=5000):
p = 0
ops = self.get_op_list()
new_ops = []
# keeping mutation_prob consistent with nasbench_101
mutation_prob = mutation_rate / (OP_SPOTS - 2)
for i, op in enumerate(ops):
if random.random() < mutation_prob:
available = [o for o in OPS if o != op]
new_ops.append(random.choice(available))
else:
new_ops.append(op)
return {'string':self.get_string_from_ops(new_ops)}
def encode_standard(self):
"""
compute the standard encoding
"""
ops = self.get_op_list()
encoding = []
for op in ops:
encoding.append(OPS.index(op))
return encoding
def get_num_params(self, nasbench):
# todo update to the newer nasbench-201 dataset
return 100
def get_paths(self):
"""
return all paths from input to output
"""
path_blueprints = [[3], [0,4], [1,5], [0,2,5]]
ops = self.get_op_list()
paths = []
for blueprint in path_blueprints:
paths.append([ops[node] for node in blueprint])
return paths
def get_path_indices(self):
"""
compute the index of each path
"""
paths = self.get_paths()
path_indices = []
for i, path in enumerate(paths):
if i == 0:
index = 0
elif i in [1, 2]:
index = NUM_OPS
else:
index = NUM_OPS + NUM_OPS ** 2
for j, op in enumerate(path):
index += OPS.index(op) * NUM_OPS ** j
path_indices.append(index)
return tuple(path_indices)
def encode_paths(self):
""" output one-hot encoding of paths """
num_paths = sum([NUM_OPS ** i for i in range(1, LONGEST_PATH_LENGTH + 1)])
path_indices = self.get_path_indices()
encoding = np.zeros(num_paths)
for index in path_indices:
encoding[index] = 1
return encoding
def path_distance(self, other):
"""
compute the distance between two architectures
by comparing their path encodings
"""
return np.sum(np.array(self.encode_paths() != np.array(other.encode_paths())))
def trunc_path_distance(self, other, cutoff=30):
"""
compute the distance between two architectures
by comparing their truncated path encodings
"""
paths = np.array(self.encode_paths()[cutoff])
other_paths = np.array(other.encode_paths()[cutoff])
return np.sum(paths != other_paths)
def edit_distance(self, other):
ops = self.get_op_list()
other_ops = other.get_op_list()
return np.sum([1 for i in range(len(ops)) if ops[i] != other_ops[i]])
def nasbot_distance(self, other):
# distance based on optimal transport between row sums, column sums, and ops
ops = self.get_op_list()
other_ops = other.get_op_list()
counts = [ops.count(op) for op in OPS]
other_counts = [other_ops.count(op) for op in OPS]
ops_dist = np.sum(np.abs(np.subtract(counts, other_counts)))
return ops_dist + self.edit_distance(other)
================================================
FILE: params.py
================================================
import sys
def algo_params(param_str):
"""
Return params list based on param_str.
These are the parameters used to produce the figures in the paper
For AlphaX and Reinforcement Learning, we used the corresponding github repos:
https://github.com/linnanwang/AlphaX-NASBench101
https://github.com/automl/nas_benchmarks
"""
params = []
if param_str == 'test':
params.append({'algo_name':'random', 'total_queries':30})
params.append({'algo_name':'evolution', 'total_queries':30})
params.append({'algo_name':'bananas', 'total_queries':30})
params.append({'algo_name':'gp_bayesopt', 'total_queries':30})
params.append({'algo_name':'dngo', 'total_queries':30})
elif param_str == 'test_simple':
params.append({'algo_name':'random', 'total_queries':30})
params.append({'algo_name':'evolution', 'total_queries':30})
elif param_str == 'random':
params.append({'algo_name':'random', 'total_queries':10})
elif param_str == 'bananas':
params.append({'algo_name':'bananas', 'total_queries':150, 'verbose':0})
elif param_str == 'main_experiments':
params.append({'algo_name':'random', 'total_queries':150})
params.append({'algo_name':'evolution', 'total_queries':150})
params.append({'algo_name':'bananas', 'total_queries':150})
params.append({'algo_name':'gp_bayesopt', 'total_queries':150})
params.append({'algo_name':'dngo', 'total_queries':150})
elif param_str == 'ablation':
params.append({'algo_name':'bananas', 'total_queries':150})
params.append({'algo_name':'bananas', 'total_queries':150, 'encoding_type':'adjacency'})
params.append({'algo_name':'gp_bayesopt', 'total_queries':150, 'distance':'path_distance'})
params.append({'algo_name':'gp_bayesopt', 'total_queries':150, 'distance':'edit_distance'})
params.append({'algo_name':'bananas', 'total_queries':150, 'acq_opt_type':'random'})
else:
print('invalid algorithm params: {}'.format(param_str))
sys.exit()
print('\n* Running experiment: ' + param_str)
return params
def meta_neuralnet_params(param_str):
if param_str == 'nasbench':
params = {'search_space':'nasbench', 'dataset':'cifar10', 'loss':'mae', 'num_layers':10, 'layer_width':20, \
'epochs':150, 'batch_size':32, 'lr':.01, 'regularization':0, 'verbose':0}
elif param_str == 'darts':
params = {'search_space':'darts', 'dataset':'cifar10', 'loss':'mape', 'num_layers':10, 'layer_width':20, \
'epochs':10000, 'batch_size':32, 'lr':.00001, 'regularization':0, 'verbose':0}
elif param_str == 'nasbench_201_cifar10':
params = {'search_space':'nasbench_201', 'dataset':'cifar10', 'loss':'mae', 'num_layers':10, 'layer_width':20, \
'epochs':150, 'batch_size':32, 'lr':.01, 'regularization':0, 'verbose':0}
elif param_str == 'nasbench_201_cifar100':
params = {'search_space':'nasbench_201', 'dataset':'cifar100', 'loss':'mae', 'num_layers':10, 'layer_width':20, \
'epochs':150, 'batch_size':32, 'lr':.01, 'regularization':0, 'verbose':0}
elif param_str == 'nasbench_201_imagenet':
params = {'search_space':'nasbench_201', 'dataset':'ImageNet16-120', 'loss':'mae', 'num_layers':10, 'layer_width':20, \
'epochs':150, 'batch_size':32, 'lr':.01, 'regularization':0, 'verbose':0}
else:
print('invalid meta neural net params: {}'.format(param_str))
sys.exit()
return params
================================================
FILE: run_experiments_parallel.sh
================================================
param_str=fifty_epochs
experiment_name=bananas
# set all instance names and zones
instances=(bananas-t4-1-vm bananas-t4-2-vm bananas-t4-3-vm bananas-t4-4-vm \
bananas-t4-5-vm bananas-t4-6-vm bananas-t4-7-vm bananas-t4-8-vm \
bananas-t4-9-vm bananas-t4-10-vm)
zones=(us-west1-b us-west1-b us-west1-b us-west1-b us-west1-b us-west1-b \
us-west1-b us-west1-b us-west1-b us-west1-b)
# set parameters based on the param string
if [ $param_str = test ]; then
start_iteration=0
end_iteration=1
k=10
untrained_filename=untrained_spec
trained_filename=trained_spec
epochs=1
fi
if [ $param_str = fifty_epochs ]; then
start_iteration=0
end_iteration=9
k=10
untrained_filename=untrained_spec
trained_filename=trained_spec
epochs=50
fi
# start bananas
for i in $(seq $start_iteration $end_iteration)
do
let start=$i*$k
let end=($i+1)*$k-1
# train the neural net
# input: all pickle files with index from 0 to i*k-1
# output: k pickle files for the architectures to train next (indices i*k to (i+1)*k-1)
echo about to run meta neural network in iteration $i
python3 metann_runner.py --experiment_name $experiment_name --params $nas_params --k $k \
--untrained_filename $untrained_filename --trained_filename $trained_filename --query $start
echo outputted architectures to train in iteration $i
# train the k architectures
let max_j=$k-1
for j in $(seq 0 $max_j )
do
let query=$i*$k+$j
instance=${instances[$j]}
zone=${zones[$j]}
untrained_filepath=$experiment_name/$untrained_filename\_$query.pkl
trained_filepath=$experiment_name/$trained_filename\_$query.pkl
echo about to copy file $untrained_filepath to instance $instance
gcloud compute scp $untrained_filepath $instance:~/naszilla/$experiment_name/ --zone $zone
echo about to ssh into instance $instance
gcloud compute ssh $instance --zone $zone --command="cd naszilla; \
python3 train_arch_runner.py --untrained_filepath $untrained_filepath \
--trained_filepath $trained_filepath --epochs $epochs" &
done
wait
echo all architectures trained in iteration $i
# copy results of trained architectures to the master CPU
let max_j=$k-1
for j in $(seq 0 $max_j )
do
let query=$i*$k+$j
instance=${instances[$j]}
zone=${zones[$j]}
trained_filepath=$experiment_name/$trained_filename\_$query.pkl
gcloud compute scp $instance:~/naszilla/$trained_filepath $experiment_name --zone $zone
done
echo finished iteration $i
done
================================================
FILE: run_experiments_sequential.py
================================================
import argparse
import time
import logging
import sys
import os
import pickle
import numpy as np
import copy
from params import *
def run_experiments(args, save_dir):
os.environ['search_space'] = args.search_space
from nas_algorithms import run_nas_algorithm
from data import Data
trials = args.trials
out_file = args.output_filename
save_specs = args.save_specs
metann_params = meta_neuralnet_params(args.search_space)
algorithm_params = algo_params(args.algo_params)
num_algos = len(algorithm_params)
logging.info(algorithm_params)
# set up search space
mp = copy.deepcopy(metann_params)
ss = mp.pop('search_space')
dataset = mp.pop('dataset')
search_space = Data(ss, dataset=dataset)
for i in range(trials):
results = []
walltimes = []
run_data = []
for j in range(num_algos):
# run NAS algorithm
print('\n* Running algorithm: {}'.format(algorithm_params[j]))
starttime = time.time()
algo_result, run_datum = run_nas_algorithm(algorithm_params[j], search_space, mp)
algo_result = np.round(algo_result, 5)
# remove unnecessary dict entries that take up space
for d in run_datum:
if not save_specs:
d.pop('spec')
for key in ['encoding', 'adjacency', 'path', 'dist_to_min']:
if key in d:
d.pop(key)
# add walltime, results, run_data
walltimes.append(time.time()-starttime)
results.append(algo_result)
run_data.append(run_datum)
# print and pickle results
filename = os.path.join(save_dir, '{}_{}.pkl'.format(out_file, i))
print('\n* Trial summary: (params, results, walltimes)')
print(algorithm_params)
print(metann_params)
print(results)
print(walltimes)
print('\n* Saving to file {}'.format(filename))
with open(filename, 'wb') as f:
pickle.dump([algorithm_params, metann_params, results, walltimes, run_data], f)
f.close()
def main(args):
# make save directory
save_dir = args.save_dir
if not os.path.exists(save_dir):
os.mkdir(save_dir)
algo_params = args.algo_params
save_path = save_dir + '/' + algo_params + '/'
if not os.path.exists(save_path):
os.mkdir(save_path)
# set up logging
log_format = '%(asctime)s %(message)s'
logging.basicConfig(stream=sys.stdout, level=logging.INFO,
format=log_format, datefmt='%m/%d %I:%M:%S %p')
fh = logging.FileHandler(os.path.join(save_dir, 'log.txt'))
fh.setFormatter(logging.Formatter(log_format))
logging.getLogger().addHandler(fh)
logging.info(args)
run_experiments(args, save_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Args for BANANAS experiments')
parser.add_argument('--trials', type=int, default=500, help='Number of trials')
parser.add_argument('--search_space', type=str, default='nasbench', \
help='nasbench or darts')
parser.add_argument('--algo_params', type=str, default='main_experiments', help='which parameters to use')
parser.add_argument('--output_filename', type=str, default='round', help='name of output files')
parser.add_argument('--save_dir', type=str, default='results_output', help='name of save directory')
parser.add_argument('--save_specs', type=bool, default=False, help='save the architecture specs')
args = parser.parse_args()
main(args)
================================================
FILE: train_arch_runner.py
================================================
import argparse
import time
import logging
import sys
import os
import pickle
sys.path.append(os.path.expanduser('~/darts/cnn'))
from train_class import Train
"""
train arch runner is used in run_experiments_parallel
- loads data by opening a pickle file containing an architecture spec
- trains that architecture for e epochs
- outputs a new pickle file with the architecture spec and its validation loss
"""
def run(args):
untrained_filepath = os.path.expanduser(args.untrained_filepath)
trained_filepath = os.path.expanduser(args.trained_filepath)
epochs = args.epochs
gpu = args.gpu
train_portion = args.train_portion
seed = args.seed
save = args.save
# load the arch spec that will be trained
dic = pickle.load(open(untrained_filepath, 'rb'))
arch = dic['spec']
print('loaded arch', arch)
# train the arch
trainer = Train()
val_accs, test_accs = trainer.main(arch,
epochs=epochs,
gpu=gpu,
train_portion=train_portion,
seed=seed,
save=save)
val_sum = 0
for epoch, val_acc in val_accs:
key = 'val_loss_' + str(epoch)
dic[key] = 100 - val_acc
val_sum += dic[key]
for epoch, test_acc in test_accs:
key = 'test_loss_' + str(epoch)
dic[key] = 100 - test_acc
val_loss_avg = val_sum / len(val_accs)
dic['val_loss_avg'] = val_loss_avg
dic['val_loss'] = 100 - val_accs[-1][-1]
dic['test_loss'] = 100 - test_accs[-1][-1]
dic['filepath'] = args.trained_filepath
print('arch {}'.format(arch))
print('val loss: {}'.format(dic['val_loss']))
print('test loss: {}'.format(dic['test_loss']))
print('val loss avg: {}'.format(dic['val_loss_avg']))
with open(trained_filepath, 'wb') as f:
pickle.dump(dic, f)
def main(args):
#set up save dir
save_dir = './'
#set up logging
log_format = '%(asctime)s %(message)s'
logging.basicConfig(stream=sys.stdout, level=logging.INFO,
format=log_format, datefmt='%m/%d %I:%M:%S %p')
fh = logging.FileHandler(os.path.join(save_dir, 'log.txt'))
fh.setFormatter(logging.Formatter(log_format))
logging.getLogger().addHandler(fh)
logging.info(args)
run(args)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Args for training a darts arch')
parser.add_argument('--untrained_filepath', type=str, default='darts_test/untrained_spec_0.pkl', help='name of input files')
parser.add_argument('--trained_filepath', type=str, default='darts_test/trained_spec_0.pkl', help='name of output files')
parser.add_argument('--epochs', type=int, default=50, help='number of training epochs')
parser.add_argument('--gpu', type=int, default=0, help='which gpu to use')
parser.add_argument('--train_portion', type=float, default=0.7, help='portion of training data used for training')
parser.add_argument('--seed', type=float, default=0, help='random seed to use')
parser.add_argument('--save', type=str, default='EXP', help='directory to save to')
args = parser.parse_args()
main(args)
