[ { "path": "LICENSE", "content": "BSD License\n-----------\n\nCopyright (c) 2016 Cong Fu, Deng Cai (http://wiki.zjulearning.org:8081/wiki/Main_Page) All rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification,\nare permitted provided that the following conditions are met:\n* Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n* Redistributions in binary form must reproduce the above copyright notice, this\n list of conditions and the following disclaimer in the documentation and/or\n other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\t\t\n" }, { "path": "Makefile", "content": "GXX=g++ -std=c++11\n#OPTM=-O3 -msse2 -msse4 -fopenmp\nOPTM=-O3 -march=native -fopenmp\nCPFLAGS=$(OPTM) -Wall -DINFO\nLDFLAGS=$(OPTM) -Wall -lboost_timer -lboost_chrono -lboost_system -DINFO\n\nINCLUDES=-I./ -I./algorithm -I./general\n\nSAMPLES=$(patsubst %.cc, %, $(wildcard samples/*.cc samples_hashing/*.cc))\nSAMPLE_OBJS=$(foreach sample, $(SAMPLES), $(sample).o)\n\nHEADERS=$(wildcard ./*.hpp ./*/*.hpp)\n\n#EFNN is currently header only, so only samples will be compiled\n\n#SHARED_LIB=libefnn.so\n#OBJS=src/efnn.o\n\nall: $(SHARED_LIB) $(SAMPLES)\n\n#$(SHARED_LIB): $(OBJS)\n#\t$(GXX) $(LDFLAGS) $(LIBS) $(OBJS) -shared -o $(SHARED_LIB)\n\n$(SAMPLES): %: %.o\n\t$(GXX) $^ -o $@ $(LDFLAGS) $(LIBS)\n\n%.o: %.cpp $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cpp -o $@\n\n%.o: %.cc $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cc -o $@\n\nclean:\n\trm -rf $(OBJS)\n\trm -rf $(SHARED_LIB)\n\trm -rf $(SAMPLES)\n\trm -rf $(SAMPLE_OBJS)\n" }, { "path": "Makefile.debug", "content": "GXX=g++ -std=c++11\nOPTM=-O2 -msse4 \nCPFLAGS=$(OPTM) -Wall -Werror -g\nLDFLAGS=$(OPTM) -Wall\n\nINCLUDES=-I./ -I./algorithm -I./general\n\nSAMPLES=$(patsubst %.cc, %, $(wildcard samples/*.cc samples_hashing/*.cc))\nSAMPLE_OBJS=$(foreach sample, $(SAMPLES), $(sample).o)\n\nHEADERS=$(wildcard ./*.hpp ./*/*.hpp) \n\n#EFNN is currently header only, so only samples will be compiled\n\n#SHARED_LIB=libefnn.so\n#OBJS=src/efnn.o\n\nall: $(SHARED_LIB) $(SAMPLES)\n\n#$(SHARED_LIB): $(OBJS)\n#\t$(GXX) $(LDFLAGS) $(LIBS) $(OBJS) -shared -o $(SHARED_LIB)\n\n$(SAMPLES): %: %.o\n\t$(GXX) $(LDFLAGS) $(LIBS) $^ -o $@\n\n%.o: %.cpp $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cpp -o $@\n\n%.o: %.cc $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cc -o $@\n\nclean:\n\trm -rf $(OBJS) \n\trm -rf $(SHARED_LIB)\n\trm -rf $(SAMPLES)\n\trm -rf $(SAMPLE_OBJS)\n\n" }, { "path": "Makefile.silent", "content": "GXX=g++ -std=c++11\n#OPTM=-O3 -msse2 -msse4 -fopenmp\nOPTM=-O3 -march=native -fopenmp\nCPFLAGS=$(OPTM) -Wall\nLDFLAGS=$(OPTM) -Wall -lboost_timer -lboost_system\n\nINCLUDES=-I./ -I./algorithm -I./general\n\nSAMPLES=$(patsubst %.cc, %, $(wildcard samples/*.cc samples_hashing/*.cc))\nSAMPLE_OBJS=$(foreach sample, $(SAMPLES), $(sample).o)\n\nHEADERS=$(wildcard ./*.hpp ./*/*.hpp)\n\n#EFNN is currently header only, so only samples will be compiled\n\n#SHARED_LIB=libefnn.so\n#OBJS=src/efnn.o\n\nall: $(SHARED_LIB) $(SAMPLES)\n\n#$(SHARED_LIB): $(OBJS)\n#\t$(GXX) $(LDFLAGS) $(LIBS) $(OBJS) -shared -o $(SHARED_LIB)\n\n$(SAMPLES): %: %.o\n\t$(GXX) $(LDFLAGS) $(LIBS) $^ -o $@\n\n%.o: %.cpp $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cpp -o $@\n\n%.o: %.cc $(HEADERS)\n\t$(GXX) $(CPFLAGS) $(INCLUDES) -c $*.cc -o $@\n\nclean:\n\trm -rf $(OBJS)\n\trm -rf $(SHARED_LIB)\n\trm -rf $(SAMPLES)\n\trm -rf $(SAMPLE_OBJS)\n" }, { "path": "README.md", "content": "EFANNA: an Extremely Fast Approximate Nearest Neighbor search Algorithm framework based on kNN graph\n============\nEFANNA is a ***flexible*** and ***efficient*** library for approximate nearest neighbor search (ANN search) on large scale data. It implements the algorithms of our paper [EFANNA : Extremely Fast Approximate Nearest Neighbor Search Algorithm Based on kNN Graph](http://arxiv.org/abs/1609.07228). \nEFANNA provides fast solutions on both ***approximate nearest neighbor graph construction*** and ***ANN search*** problems. \nEFANNA is also flexible to adopt all kinds of hierarchical structure for initialization, such as random projection tree, hierarchical clustering tree, [multi-table hashing](https://github.com/fc731097343/efanna/tree/master/samples_hashing) and so on. \n\nWhat's new\n-------\n+ **Please see our more advanced search algorithm [NSG](https://github.com/ZJULearning/nsg)** Jan 13, 2018\n+ **The [paper](http://arxiv.org/abs/1609.07228) updated significantly.** Dec 6, 2016\n+ **Algorithm improved and AVX instructions supported.** Nov 30, 2016 \n+ **Parallelism with OpenMP.** Sep 26, 2016 \n\nBenchmark data set\n-------\n* [SIFT1M and GIST1M](http://corpus-texmex.irisa.fr/)\n\nANN search performance\n------\nThe performance was tested without parallelism. \n\n![SIFT1nn](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/SIFT_1nn.png) \n![SIFT100nn](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/SIFT_100nn.png) \n![GIST1nn](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/GIST_1nn.png) \n![GIST100nn](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/GIST_100nn.png) \nCompared Algorithms: \n* [kGraph](http://www.kgraph.org) \n* [flann](http://www.cs.ubc.ca/research/flann/) \n* [IEH](http://ieeexplore.ieee.org/document/6734715/) : Fast and accurate hashing via iterative nearest neighbors expansion \n* [GNNS](https://webdocs.cs.ualberta.ca/~abbasiya/gnns.pdf) : Fast Approximate Nearest-Neighbor Search with k-Nearest Neighbor Graph \n\nkNN Graph Construction Performance\n------\nThe performance was tested without parallelism. \n\n![SIFT1nnGraph](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/SIFT_graph.png) \n![SIFT100nnGraph](http://www.cad.zju.edu.cn/home/dengcai/Data/Hashing/GIST_graph.png) \n\nCompared Algorithms: \n* [Kgraph](http://www.kgraph.org) (same with NN-descent) \n* [NN-expansion](https://webdocs.cs.ualberta.ca/~abbasiya/gnns.pdf) (same with GNNS) \n* [SGraph](http://ieeexplore.ieee.org/document/6247790/) : Scalable k-NN graph construction for visual descriptors \n* [FastKNN](http://link.springer.com/chapter/10.1007/978-3-642-40991-2_42) : Fast kNN Graph Construction with Locality Sensitive Hashing \n* [LargeVis](http://dl.acm.org/citation.cfm?id=2883041) : Visualizing Large-scale and High-dimensional Data \n\nHow To Complie \n-------\nGo to the root directory of EFANNA and make. \n\n\tcd efanna/\n\tmake\n\nHow To Use \n------\nEFANNA uses a composite index to carry out ANN search, which includes an approximate kNN graph and a number of tree structures. They can be built by this library as a whole or seperately. \n \nYou may build the kNN graph seperately for other use, like other graph based machine learning algorithms. \n \n Below are some demos. \n\n* kNN graph building : \n\n\t\tcd efanna/samples/ \n\t\t./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 8 30 25 10 10 \n\t\t \n Meaning of the parameters(from left to right): \n\n\tsift_base.fvecs -- database points \n\tsift.graph -- graph built by EFANNA \n\t\n\t8 -- number of trees used to build the graph (larger is more accurate but slower) \n\t8 -- conquer-to-depeth(smaller is more accurate but slower) \n\t8 -- number of iterations to build the graph \n\t \n\t30 -- L (larger is more accurate but slower, no smaller than K) \n\t25 -- check (larger is more accurate but slower, no smaller than K) \n\t10 -- K, for KNN graph \n\t10 -- S (larger is more accurate but slower)\n\n\t\n* tree building : \n \n\t\tcd efanna/samples/\n\t\t./efanna_index_buildtrees sift_base.fvecs sift.trees 32\n \n Meaning of the parameters(from left to right): \n \n\tsift_base.fvecs -- database points \n\tsift.trees -- struncated KD-trees built by EFANNA \n\t32 -- number of trees to build \n\n\n* index building at one time: \n\n\t\tcd efanna/samples/\n\t\t./efanna_index_buildall sift_base.fvecs sift.graph sift.trees 32 8 8 200 200 100 10 8\n \n Meaning of the parameters(from left to right) \n\t\n\tsift_base.fvecs -- database points \n\tsift.trees -- struncated KD-trees built by EFANNA \n\tsift.graph -- approximate KNN graph built by EFANNA\n\t32 -- number of trees in total for building index \n\t8 -- conquer-to-depth \n\t8 -- iteration number \n\t200 -- L (larger is more accurate but slower, no smaller than K) \n\t200 -- check (larger is more accurate but slower, no smaller than K) \n\t100 -- K, for KNN graph\n\t10 -- S (larger is more accurate but slower) \n\t8 -- 8 out of 32 trees are used for building graph\n\n \n* ANN search\n\n\t\tcd efanna/samples/\n\t\t./efanna_search sift_base.fvecs sift.trees sift.graph sift_query.fvecs sift.results 16 4 1200 200 10\n \n Meaning of the parameters(from left to right): \n \n\tsift_base.fvecs -- database points \n\tsift.trees -- prebuilt struncated KD-trees used for search \n\tsift.graph -- prebuilt kNN graph \n\tsift_query -- sift query points \n\tsift.results -- path to save ANN search results of given query \n\t16 -- number of trees to use (no greater than the number of prebuilt trees) \n\t4 -- number of epoches \n\t1200 -- pool size factor (larger is more accurate but slower, usually 6~10 times larger than extend factor) \n\t200 -- extend factor (larger is more accurate but slower) \n\t10 -- required number of returned neighbors (i.e. k of k-NN) \n \n \n* Evaluation\n\n\t\tcd efanna/samples/\n\t\t./evaluate sift.results sift_groundtruth.ivecs 10\n\n Meaning of the parameters(from left to right): \n \n\tsift.results -- search results file \n\tsift_groundtruth.ivecs -- ground truth file \n\t10 -- evaluate the 10NN accuracy (the only first 10 points returned by the algorithm are examined, how many points are among the true 10 nearest neighbors of the query) \n\n\nSee our paper or user manual for more details about the parameters and interfaces.\n\nOutput format\n------\nThe file format of approximate kNN graph and ANN search results are the same. \nSuppose the database has N points, and numbered from 0 to N-1. You want to build an approximate kNN graph. The graph can be regarded as a N * k Matrix. The saved kNN graph binary file saves the matrix by row. The first 4 bytes of each row saves the int value of k, next 4 bytes saves the value of M and next 4 bytes saves the float value of the norm of the point. Then it follows k*4 bytes, saving the indices of the k nearest neighbors of respective point. The N rows are saved continuously without seperating characters. \n\nSimilarly, suppose the query data has n points, numbered 0 to n-1. You want EFANNA to return k nearest neighbors for each query. The result file will save n rows like the graph file. It saves the returned indices row by row. Each row starts with 4 bytes recording value of k, and follows k*4 bytes recording neighbors' indices. \n\nInput of EFANNA\n------\nBecause there is no unified format for input data, users may need to write input function to read your own data. You may imitate the input function in our sample code (sample/efanna\\_efanna\\_index\\_buildgraph.cc) to load the data into our matrix.\n\nTo use SIMD instruction optimization, you should pay attention to the data alignment problem of SSE / AVX instruction. \n\nCompare with EFANNA without parallelism and SSE/AVX instructions\n------\nTo disable the parallelism, there is no need to modify the code. Simply\n\n export OMP_NUM_THREADS=1\n\nbefore you run the code. Then the code will only use one thread. This is a very convenient way to control the number of threads used.\n\nTo disable SSE/AVX instructions, you need to modify samples/xxxx.cc, find the line\n\n FIndex index(dataset, new L2DistanceAVX(), efanna::KDTreeUbIndexParams(true, trees ,mlevel ,epochs,checkK,L, kNN, trees, S));\n\nChange **L2DistanceAVX** to **L2Distance** and build the project. Now the SSE/AVX instructions are disabled.\nIf you want to try SSE instead of AVX, try **L2DistanceSSE**\n\nParameters to get the Fig. 4/5 (10-NN approximate graph construction) in our paper \n------\nSIFT1M:\n\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 0 20 10 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 1 20 10 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 2 20 10 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 3 20 10 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 5 20 10 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 6 20 20 10 10\n ./efanna_index_buildgraph sift_base.fvecs sift.graph 8 8 6 20 30 10 10\n\nGIST1M:\n\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 2 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 3 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 4 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 5 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 6 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 7 30 30 10 10\n ./efanna_index_buildgraph gist_base.fvecs gist.graph 8 8 10 30 40 10 10\n\nAcknowledgment\n------\nOur code framework imitates [Flann](http://www.cs.ubc.ca/research/flann/) to make it scalable, and the implemnetation of NN-descent is taken from [Kgraph](http://www.kgraph.org). They proposed the NN-descent algorithm. Many thanks to them for inspiration.\n\nWhat to do\n-------\n* Add more initial algorithm choice\t\n" }, { "path": "algorithm/base_index.hpp", "content": "#ifndef EFANNA_BASE_INDEX_H_\n#define EFANNA_BASE_INDEX_H_\n\n#include \"general/params.hpp\"\n#include \"general/distance.hpp\"\n#include \"general/matrix.hpp\"\n#include \n#include \n#include \n#include \n#include \n#include \n#include \"boost/smart_ptr/detail/spinlock.hpp\"\n#include \n\n\n\n//#define BATCH_SIZE 200\n\nnamespace efanna{\n\n typedef boost::detail::spinlock Lock;\n typedef std::lock_guard LockGuard;\n struct Point {\n unsigned id;\n float dist;\n bool flag;\n Point () {}\n Point (unsigned i, float d, bool f = true): id(i), dist(d), flag(f) {\n }\n bool operator < (const Point &n) const{\n return this->dist < n.dist;\n }\n };\n\n typedef std::vector Points;\n static inline unsigned InsertIntoKnn (Point *addr, unsigned K, Point nn) {\n // find the location to insert\n unsigned j;\n unsigned i = K;\n while (i > 0) {\n j = i - 1;\n if (addr[j].dist <= nn.dist) break;\n i = j;\n }\n // check for equal ID\n unsigned l = i;\n while (l > 0) {\n j = l - 1;\n if (addr[j].dist < nn.dist) break;\n if (addr[j].id == nn.id) return K + 1;\n l = j;\n }\n // i <= K-1\n j = K;\n while (j > i) {\n addr[j] = addr[j-1];\n --j;\n }\n addr[i] = nn;\n return i;\n }\n struct Neighbor {\n std::shared_ptr lock;\n float radius;\n float radiusM;\n Points pool;\n unsigned L;\n unsigned Range;\n bool found;\n std::vector nn_old;\n std::vector nn_new;\n std::vector rnn_old;\n std::vector rnn_new;\n\n Neighbor() : lock(std::make_shared())\n {\n }\n\n unsigned insert (unsigned id, float dist) {\n if (dist > radius) return pool.size();\n LockGuard guard(*lock);\n unsigned l = InsertIntoKnn(&pool[0], L, Point(id, dist, true));\n if (l <= L) {\n if (L + 1 < pool.size()) {\n ++L;\n }\n else {\n radius = pool[L-1].dist;\n }\n }\n return l;\n }\n\n template \n void join (C callback) const {\n for (unsigned const i: nn_new) {\n for (unsigned const j: nn_new) {\n if (i < j) {\n callback(i, j);\n }\n }\n for (unsigned j: nn_old) {\n callback(i, j);\n }\n }\n }\n\n };\n\n template \n class InitIndex{\n public:\n InitIndex(const Matrix& features, const Distance* d, const IndexParams& params):\n features_(features),\n distance_(d),\n params_(params)\n {\n }\n virtual ~InitIndex() {};\n virtual void buildTrees(){}\n\n virtual void buildIndex()\n {\n\t buildIndexImpl();\n }\n virtual void buildIndexImpl() = 0;\n virtual void loadIndex(char* filename) = 0;\n virtual void saveIndex(char* filename) = 0;\n virtual void loadTrees(char* filename) = 0;\n virtual void saveTrees(char* filename) = 0;\n virtual void loadGraph(char* filename) = 0;\n virtual void saveGraph(char* filename) = 0;\n\tvirtual void outputVisitBucketNum() = 0;\n\n void saveResults(char* filename){\n std::ofstream out(filename,std::ios::binary);\n std::vector>::iterator i;\n//std::cout<::iterator j;\n int dim = i->size();\n//std::cout<begin(); j != i->end(); j++){\n int id = *j;\n out.write((char*)&id, sizeof(int));\n }\n }\n out.close();\n }\nSearchParams SP;\n void setSearchParams(int epochs, int init_num, int extend_to,int search_trees, int search_lv, int search_method){\n SP.search_epoches = epochs;\n SP.search_init_num = init_num;\n if(extend_to>init_num) SP.extend_to = init_num;\n else SP.extend_to = extend_to;\n SP.search_depth = search_lv;\n SP.tree_num = search_trees;\n SP.search_method = search_method;\n }\n\n\n\n void nnExpansion_kgraph(size_t K, const DataType* qNow, std::vector& pool, std::vector& results){\n\n unsigned int base_n = features_.get_rows();\n boost::dynamic_bitset<> tbflag(base_n, false);\n boost::dynamic_bitset<> newflag(base_n, false);\n\n std::vector knn(K + SP.extend_to +1);\n\n int remainder = SP.search_init_num % SP.extend_to;\n int nSeg = SP.search_init_num / SP.extend_to;\n\n //clock_t s,f;\n\n int Iter = nSeg;\n if (remainder > 0) Iter++;\n int Jter = SP.extend_to;\n\n\n for(int i = 0; i 0) && (i == Iter-1)) Jter=remainder;\n\n unsigned int L = 0;\n for(int j=0; j compare(qNow, features_.get_row(knn[k].id), features_.get_cols());\n newflag.set(knn[k].id);\n }\n std::sort(knn.begin(), knn.begin() + L);\n\n //s = clock();\n unsigned int k = 0;\n while (k < L) {\n unsigned int nk = L;\n if (newflag.test(knn[k].id)){\n newflag.reset(knn[k].id);\n tbflag.set(knn[k].id);\n typename CandidateHeap::reverse_iterator neighbor = knn_graph[knn[k].id].rbegin();\n for(size_t nnk = 0;nnk < params_.K && neighbor != knn_graph[knn[k].id].rend(); neighbor++, nnk++){\n if(tbflag.test(neighbor->row_id))continue;\n tbflag.set(neighbor->row_id);\n newflag.set(neighbor->row_id);\n float dist = distance_->compare(qNow, features_.get_row(neighbor->row_id), features_.get_cols());\n Point nn(neighbor->row_id, dist);\n unsigned int r = InsertIntoKnn(&knn[0], L, nn);\n \t if ( (r <= L) && (L + 1 < knn.size())) ++L;\n if (r < nk) nk = r;\n }\n }\n if (nk <= k) k = nk;\n else ++k;\n }\n //f = clock();\n //sum = sum + f-s;\n\n\n if (L > K) L = K;\n if (results.empty()) {\n results.reserve(K + 1);\n results.resize(L + 1);\n std::copy(knn.begin(), knn.begin() + L, results.begin());\n }\n else {\n for (unsigned int l = 0; l < L; ++l) {\n unsigned r = InsertIntoKnn(&results[0], results.size() - 1, knn[l]);\n if (r < results.size() /* inserted */ && results.size() < (K + 1)) {\n results.resize(results.size() + 1);\n }\n }\n }\n }\n results.pop_back();\n }\n\n void nnExpansion(size_t K, const DataType* qNow, std::vector& pool, std::vector& res){\n\n unsigned int base_n = features_.get_rows();\n boost::dynamic_bitset<> tbflag(base_n, false);\n boost::dynamic_bitset<> newflag(base_n, false);\n\n CandidateHeap Candidates;\n\n int remainder = SP.search_init_num % SP.extend_to;\n int nSeg = SP.search_init_num / SP.extend_to;\n\n int segIter = nSeg;\n if (remainder > 0) segIter++;\n int Jter = SP.extend_to;\n\n CandidateHeap Results;\n\n for(int seg = 0; seg 0) && (seg == segIter-1)) Jter=remainder;\n\n for(int j=0; j =base_n) std::cout << \"query:\" << cur << \" Init \"<< nn << std::endl;\n if(!tbflag.test(nn)){\n newflag.set(nn);\n Candidate c(nn, distance_->compare(qNow, features_.get_row(nn), features_.get_cols()));\n Candidates.insert(c);\n }\n }\n\n std::vector ids;\n int iter=0;\n while(iter++ < SP.search_epoches){\n //the heap is max heap\n ids.clear();\n typename CandidateHeap::reverse_iterator it = Candidates.rbegin();\n for(unsigned j = 0; j < SP.extend_to && it != Candidates.rend(); j++,it++){\n \t// if(it->row_id>=base_n) std::cout<<\"query:\"<< cur<<\" Judge node \"<row_id<row_id)){\n newflag.reset(it->row_id);\n typename CandidateHeap::reverse_iterator neighbor = knn_graph[it->row_id].rbegin();\n for(; neighbor != knn_graph[it->row_id].rend(); neighbor++){\n \t// if(neighbor->row_id>=base_n) std::cout<<\"query:\"<< cur<<\" Judge neighbor \"<row_id<row_id))continue;\n tbflag.set(neighbor->row_id);\n ids.push_back(neighbor->row_id);\n }\n }\n }\n for(size_t j = 0; j < ids.size(); j++){\n Candidate c(ids[j], distance_->compare(qNow, features_.get_row(ids[j]), features_.get_cols()) );\n Candidates.insert(c);\n newflag.set(ids[j]);\n if(Candidates.size() > (unsigned int)SP.extend_to)Candidates.erase(Candidates.begin());\n }\n }\n typename CandidateHeap::reverse_iterator it = Candidates.rbegin();\n for(unsigned int j = 0; j < K && it != Candidates.rend(); j++,it++){\n Results.insert(*it);\n if(Results.size() > K)Results.erase(Results.begin());\n }\n }\n typename CandidateHeap::reverse_iterator it = Results.rbegin();\n for(unsigned int j = 0; j < K && it != Candidates.rend(); j++,it++){\n \t res.push_back(it->row_id);\n }\n }\n\n virtual void knnSearch(int K, const Matrix& query){\n getNeighbors(K,query);\n }\n virtual void getNeighbors(size_t K, const Matrix& query) = 0;\n virtual void initGraph() = 0;\n\n\n\n\n\n\n //std::vector Range;\n std::vector > graph;\n\n\n\n std::vector nhoods;\n void join(){\n size_t dim = features_.get_cols();\n size_t cc = 0;\n#pragma omp parallel for default(shared) schedule(dynamic, 100) reduction(+:cc)\n for(size_t i = 0; i < nhoods.size(); i++){\n size_t uu = 0;\n nhoods[i].found = false;\n/*\n for(size_t newi = 0; newi < nhoods[i].nn_new.size(); newi++){\n for(size_t newj = newi+1; newj < nhoods[i].nn_new.size(); newj++){\n unsigned a = nhoods[i].nn_new[newi];\n unsigned b = nhoods[i].nn_new[newj];\n DataType dist = distance_->compare(\n features_.get_row(a), features_.get_row(b), dim);\n unsigned r = nhoods[a].insert(b,dist);\n if(r < params_.Check_K){uu += 2;}\n nhoods[b].insert(a,dist);\n }\n\n for(size_t oldj = 0; oldj < nhoods[i].nn_old.size(); oldj++){\n unsigned a = nhoods[i].nn_new[newi];\n unsigned b = nhoods[i].nn_old[oldj];\n DataType dist = distance_->compare(\n features_.get_row(a), features_.get_row(b), dim);\n unsigned r = nhoods[a].insert(b,dist);\n if(r < params_.Check_K){uu += 2;}\n nhoods[b].insert(a,dist);\n }\n }\n*/\n\tnhoods[i].join([&](unsigned i, unsigned j) {\n\t\tDataType dist = distance_->compare(\n \t\tfeatures_.get_row(i), features_.get_row(j), dim);\n\t\t++cc;\n\t\tunsigned r;\n\t\tr = nhoods[i].insert(j, dist);\n\t\tif (r < params_.Check_K) ++uu;\n\t\tnhoods[j].insert(i, dist);\n\t\tif (r < params_.Check_K) ++uu;\n\t});\n nhoods[i].found = uu > 0;\n }\n\n }\n\n void update (int paramL) {\n for (size_t i = 0; i < nhoods.size(); i++) {\n nhoods[i].nn_new.clear();\n nhoods[i].nn_old.clear();\n nhoods[i].rnn_new.clear();\n nhoods[i].rnn_old.clear();\n nhoods[i].radius = nhoods[i].pool.back().dist;\n }\n //find longest new\n#pragma omp parallel for\n for(size_t i = 0; i < nhoods.size(); i++){\n if(nhoods[i].found){\n unsigned maxl = nhoods[i].Range + params_.S < nhoods[i].L ? nhoods[i].Range + params_.S : nhoods[i].L;\n unsigned c = 0;\n unsigned l = 0;\n while ((l < maxl) && (c < params_.S)) {\n if (nhoods[i].pool[l].flag) ++c;\n ++l;\n }\n nhoods[i].Range = l;\n }\n nhoods[i].radiusM = nhoods[i].pool[nhoods[i].Range-1].dist;\n }\n#pragma omp parallel for\n for (unsigned n = 0; n < nhoods.size(); ++n) {\n Neighbor &nhood = nhoods[n];\n std::vector &nn_new = nhood.nn_new;\n std::vector &nn_old = nhood.nn_old;\n for (unsigned l = 0; l < nhood.Range; ++l) {\n Point &nn = nhood.pool[l];\n Neighbor &nhood_o = nhoods[nn.id]; // nhood on the other side of the edge\n if (nn.flag) {\n nn_new.push_back(nn.id);\n if (nn.dist > nhood_o.radiusM) {\n LockGuard guard(*nhood_o.lock);\n nhood_o.rnn_new.push_back(n);\n }\n nn.flag = false;\n }\n else {\n nn_old.push_back(nn.id);\n if (nn.dist > nhood_o.radiusM) {\n LockGuard guard(*nhood_o.lock);\n nhood_o.rnn_old.push_back(n);\n }\n }\n }\n }\n for (unsigned i = 0; i < nhoods.size(); ++i) {\n std::vector &nn_new = nhoods[i].nn_new;\n std::vector &nn_old = nhoods[i].nn_old;\n std::vector &rnn_new = nhoods[i].rnn_new;\n std::vector &rnn_old = nhoods[i].rnn_old;\n if (paramL && (rnn_new.size() > (unsigned int)paramL)) {\n random_shuffle(rnn_new.begin(), rnn_new.end());\n rnn_new.resize(paramL);\n }\n nn_new.insert(nn_new.end(), rnn_new.begin(), rnn_new.end());\n if (paramL && (rnn_old.size() > (unsigned int)paramL)) {\n random_shuffle(rnn_old.begin(), rnn_old.end());\n rnn_old.resize(paramL);\n }\n nn_old.insert(nn_old.end(), rnn_old.begin(), rnn_old.end());\n }\n }\n void refineGraph(){\n std::cout << \" refineGraph\" << std::endl;\n int iter = 0;\n clock_t s,f;\n s = clock();unsigned int l=100;\n while(iter++ < params_.build_epoches){\n join();//std::cout<<\"after join\"< c(nhoods[i].pool[j].id,nhoods[i].pool[j].dist);\n can.insert(c);\n\t \n }\n while(can.size()compare(features_.get_row(i), features_.get_row(id),features_.get_cols());\n Candidate c(id, dist);\n can.insert(c);\n }\n knn_graph.push_back(can);\n }\n*/\n\tg.resize(nhoods.size());\n\tM.resize(nhoods.size());\n\tgs.resize(nhoods.size());\n\tfor(unsigned i = 0; i < nhoods.size();i++){\n\t M[i] = nhoods[i].Range;\n\t g[i].resize(nhoods[i].pool.size());\n\t std::copy(nhoods[i].pool.begin(), nhoods[i].pool.end(), g[i].begin());\n\t gs[i].resize(params_.K);\n\t for(unsigned j = 0; j < params_.K;j++)\n\t\tgs[i][j] = g[i][j].id;\n\t}\n\t\t\n }\n \n void calculate_norm(){\n\tunsigned N = features_.get_rows();\n\tunsigned D = features_.get_cols();\n\tnorms.resize(N);\n#pragma omp parallel for\n for (unsigned n = 0; n < N; ++n) {\n\t norms[n] = distance_->norm(features_.get_row(n),D);\n\t}\n }\ntypedef std::set, std::greater> > CandidateHeap;\ntypedef std::vector IndexVec;\n\tsize_t getGraphSize(){return gs.size();}\n\tstd::vector getGraphRow(unsigned row_id){\n\t\tstd::vector row;\n\t\tif(gs.size() > row_id){\n\t\t\tfor(unsigned i = 0; i < gs[row_id].size(); i++)row.push_back(gs[row_id][i]);\n\t\t}\n\t\treturn row;\n\t}\n\n\nprotected:\n const Matrix features_;\n const Distance* distance_;\n const IndexParams params_;\n std::vector > knn_table_gt;\n std::vector > g;\n std::vector > gs;\n std::vector M;\n //std::vector> knn_graph;\n std::vector knn_graph;\n std::vector norms;\n std::vector > nn_results;\n DataType* Radius;\n };\n#define USING_BASECLASS_SYMBOLS \\\n using InitIndex::distance_;\\\n using InitIndex::params_;\\\n using InitIndex::features_;\\\n using InitIndex::buildIndex;\\\n using InitIndex::knn_table_gt;\\\n using InitIndex::nn_results;\\\n using InitIndex::saveResults;\\\n using InitIndex::knn_graph;\\\n using InitIndex::refineGraph;\\\n using InitIndex::nhoods;\\\n using InitIndex::SP;\\\n using InitIndex::nnExpansion;\\\n using InitIndex::nnExpansion_kgraph;\\\n using InitIndex::g;\\\n using InitIndex::gs;\\\n using InitIndex::M;\\\n using InitIndex::norms;\n\n}\n#endif\n" }, { "path": "algorithm/hashing_index.hpp", "content": "#ifndef EFANNA_HASHING_INDEX_H_\n#define EFANNA_HASHING_INDEX_H_\n\n#include \"algorithm/base_index.hpp\"\n#include \n#include \n//for Debug\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n//#define MAX_RADIUS 6\n\n\nnamespace efanna{\nstruct HASHINGIndexParams : public IndexParams\n{\n HASHINGIndexParams(int codelen, int TableNum,int UpperBits, int HashRadius, char*& BaseCodeFile, char*& QueryCodeFile, int codelenShift = 0)\n {\n init_index_type = HASHING;\n ValueType len;\n len.int_val = codelen;\n extra_params.insert(std::make_pair(\"codelen\",len));\n ValueType nTab;\n nTab.int_val = TableNum;\n extra_params.insert(std::make_pair(\"tablenum\",nTab));\n ValueType upb;\n upb.int_val = UpperBits;\n extra_params.insert(std::make_pair(\"upbits\",upb));\n ValueType radius;\n radius.int_val = HashRadius;\n extra_params.insert(std::make_pair(\"radius\",radius));\n ValueType bcf;\n bcf.str_pt = BaseCodeFile;\n extra_params.insert(std::make_pair(\"bcfile\",bcf));\n ValueType qcf;\n qcf.str_pt = QueryCodeFile;\n extra_params.insert(std::make_pair(\"qcfile\",qcf));\n ValueType lenShift;\n lenShift.int_val = codelenShift;\n extra_params.insert(std::make_pair(\"lenshift\",lenShift));\n }\n};\n\ntemplate \nclass HASHINGIndex : public InitIndex\n{\n public:\n\n typedef InitIndex BaseClass;\n\n typedef std::vector Codes;\n typedef std::unordered_map > HashBucket;\n typedef std::vector HashTable;\n\n typedef std::vector Codes64;\n typedef std::unordered_map > HashBucket64;\n typedef std::vector HashTable64;\n\n\n HASHINGIndex(const Matrix& dataset, const Distance* d, const IndexParams& params = HASHINGIndexParams(0,NULL,NULL)) :\n BaseClass(dataset,d,params)\n {\n std::cout<<\"HASHING initial, max code length : 64\" <second).int_val;\n std::cout << \"use \"<second).int_val;\n int actuallen = codelength - codelengthshift;\n if(actuallen > 0){\n std::cout << \"Actually use \"<< actuallen<< \" bit code\"<< std::endl;\n }else{\n std::cout << \"lenShift error: could not be larger than the code length! \"<< std::endl;\n }\n }\n else{\n codelengthshift = 0;\n }\n\n it = params_.extra_params.find(\"tablenum\");\n if(it != params_.extra_params.end()){\n tablenum = (it->second).int_val;\n std::cout << \"use \"<second).int_val;\n std::cout << \"use upper \"<= codelength-codelengthshift){\n std::cout << \"upbits should be smaller than the actual codelength!\" << std::endl;\n return;\n }\n\n\n int actuallen = codelength - codelengthshift;\n it = params_.extra_params.find(\"radius\");\n if(it != params_.extra_params.end()){\n radius = (it->second).int_val;\n if(actuallen<=32){\n if(radius > 13){\n std::cout << \"radius greater than 13 not supported yet!\" << std::endl;\n radius = 13;\n }\n }else if(actuallen<=36){\n if(radius > 11){\n std::cout << \"radius greater than 11 not supported yet!\" << std::endl;\n radius = 11;\n }\n }else if(actuallen<=40){\n if(radius > 10){\n std::cout << \"radius greater than 10 not supported yet!\" << std::endl;\n radius = 10;\n }\n }else if(actuallen<=48){\n if(radius > 9){\n std::cout << \"radius greater than 9 not supported yet!\" << std::endl;\n radius = 9;\n }\n }else if(actuallen<=60){\n if(radius > 8){\n std::cout << \"radius greater than 8 not supported yet!\" << std::endl;\n radius = 8;\n }\n }else{ //actuallen<=64\n if(radius > 7){\n std::cout << \"radius greater than 7 not supported yet!\" << std::endl;\n radius = 7;\n }\n }\n std::cout << \"search hamming radius \"<second).str_pt;\n std::string str(fpath);\n std::cout << \"Loading base code from \" << str << std::endl;\n\n if (codelength <= 32 ){\n LoadCode32(fpath, BaseCode);\n }else if(codelength <= 64 ){\n LoadCode64(fpath, BaseCode64);\n }else{\n std::cout<<\"code length not supported yet!\"<second).str_pt;\n std::string str(fpath);\n std::cout << \"Loading query code from \" << str << std::endl;\n\n if (codelength <= 32 ){\n LoadCode32(fpath, QueryCode);\n }else if(codelength <= 64 ){\n LoadCode64(fpath, QueryCode64);\n }else{\n std::cout<<\"code length not supported yet!\"<& baseAll){\n if (tablenum < 1){\n std::cout<<\"Total hash table num error! \"<> codeFile;\n ss.clear();\n\n std::ifstream in(codeFile.c_str(), std::ios::binary);\n if(!in.is_open()){std::cout<<\"open file \" << filename <<\" error\"<< std::endl;return;}\n\n int codeNum;\n in.read((char*)&codeNum,4);\n if (codeNum != 1){\n std::cout<<\"Codefile \"<< j << \" error!\"<> codelengthshift;\n if (codetmp > maxValue){\n std::cout<<\"codetmp: \"<< codetmp <& baseAll){\n if (tablenum < 1){\n std::cout<<\"Total hash table num error! \"<> codeFile;\n ss.clear();\n\n std::ifstream in(codeFile.c_str(), std::ios::binary);\n if(!in.is_open()){std::cout<<\"open file \" << filename <<\" error\"<< std::endl;return;}\n\n int codeNum;\n in.read((char*)&codeNum,4);\n if (codeNum != 1){\n std::cout<<\"Codefile \"<< j << \" error!\"<> codelengthshift;\n if (codetmp > maxValue){\n std::cout<<\"codetmp: \"<< codetmp <& baseAll ,std::vector& tbAll){\n\n for(size_t h=0; h < baseAll.size(); h++){\n Codes& base = baseAll[h];\n\n HashTable tb;\n for(int i = 0; i < (1 << upbits); i++){\n HashBucket emptyBucket;\n tb.push_back(emptyBucket);\n }\n\n for(size_t i = 0; i < base.size(); i ++){\n unsigned int idx1 = base[i] >> lowbits;\n unsigned int idx2 = base[i] - (idx1 << lowbits);\n if(tb[idx1].find(idx2) != tb[idx1].end()){\n tb[idx1][idx2].push_back(i);\n }else{\n std::vector v;\n v.push_back(i);\n tb[idx1].insert(make_pair(idx2,v));\n }\n }\n tbAll.push_back(tb);\n }\n }\n\n void generateMask32(){\n //i = 0 means the origin code\n HammingBallMask.push_back(0);\n HammingRadius.push_back(HammingBallMask.size());\n\n if(radius>0){\n //radius 1\n for(int i = 0; i < codelength; i++){\n unsigned int mask = 1 << i;\n HammingBallMask.push_back(mask);\n }\n HammingRadius.push_back(HammingBallMask.size());\n }\n\n if(radius>1){\n //radius 2\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n unsigned int mask = (1<2){\n //radius 3\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n unsigned int mask = (1<3){\n //radius 4\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n unsigned int mask = (1<4){\n //radius 5\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n unsigned int mask = (1<5){\n //radius 6\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n unsigned int mask = (1<6){\n //radius 7\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n unsigned int mask = (1<7){\n //radius 8\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n unsigned int mask = (1<8){\n //radius 9\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n unsigned int mask = (1<9){\n //radius 10\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n unsigned int mask = (1<10){\n //radius 11\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n for(int h = g+1; h < codelength; h++){\n unsigned int mask = (1<11){\n //radius 12\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n for(int h = g+1; h < codelength; h++){\n for(int l = h+1; h < codelength; l++){\n unsigned int mask = (1<12){\n //radius 13\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n for(int h = g+1; h < codelength; h++){\n for(int l = h+1; h < codelength; l++){\n for(int m = l+1; m < codelength; m++){\n unsigned int mask = (1<& baseAll ,std::vector& tbAll){\n\n for(size_t h=0; h < baseAll.size(); h++){\n Codes64& base = baseAll[h];\n\n HashTable64 tb;\n for(int i = 0; i < (1 << upbits); i++){\n HashBucket64 emptyBucket;\n tb.push_back(emptyBucket);\n }\n\n for(size_t i = 0; i < base.size(); i ++){\n unsigned int idx1 = base[i] >> lowbits;\n unsigned long idx2 = base[i] - ((unsigned long)idx1 << lowbits);\n if(tb[idx1].find(idx2) != tb[idx1].end()){\n tb[idx1][idx2].push_back(i);\n }else{\n std::vector v;\n v.push_back(i);\n tb[idx1].insert(make_pair(idx2,v));\n }\n }\n tbAll.push_back(tb);\n }\n }\n\n\n\n void generateMask64(){\n //i = 0 means the origin code\n HammingBallMask64.push_back(0);\n HammingRadius.push_back(HammingBallMask64.size());\n\n unsigned long One = 1;\n if(radius>0){\n //radius 1\n for(int i = 0; i < codelength; i++){\n unsigned long mask = One << i;\n HammingBallMask64.push_back(mask);\n }\n HammingRadius.push_back(HammingBallMask64.size());\n }\n\n if(radius>1){\n //radius 2\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n unsigned long mask = (One<2){\n //radius 3\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n unsigned long mask = (One<3){\n //radius 4\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n unsigned long mask = (One<4){\n //radius 5\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n unsigned long mask = (One<5){\n //radius 6\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n unsigned long mask = (One<6){\n //radius 7\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n unsigned long mask = (One<7){\n //radius 8\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n unsigned long mask = (One<8){\n //radius 9\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n unsigned long mask = (One<9){\n //radius 10\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n unsigned long mask = (One<10){\n //radius 11\n for(int i = 0; i < codelength; i++){\n for(int j = i+1; j < codelength; j++){\n for(int k = j+1; k < codelength; k++){\n for(int a = k+1; a < codelength; a++){\n for(int b = a+1; b < codelength; b++){\n for(int c = b+1; c < codelength; c++){\n for(int d = c+1; d < codelength; d++){\n for(int e = d+1; e < codelength; e++){\n for(int f = e+1; f < codelength; f++){\n for(int g = f+1; g < codelength; g++){\n for(int h = g+1; h < codelength; h++){\n unsigned long mask = (One<& query){\n\n if(gs.size() != features_.get_rows()){\n if (codelength <= 32 ){\n getNeighbors32(K,query);\n }else if(codelength <= 64 ){\n getNeighbors64(K,query);\n }else{\n std::cout<<\"code length not supported yet!\"<& query){\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< tbflag(features_.get_rows(), false);\n\n nn_results.clear();\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n tbflag.reset();\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(unsigned int h=0; h < QueryCode.size(); h++){\n unsigned int searchcode = QueryCode[h][cur] ^ HammingBallMask[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned int idx2 = searchcode - (idx1 << lowbits);\n\n HashBucket::iterator bucket= htb[h][idx1].find(idx2);\n if(bucket != htb[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n if (p> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n std::partial_sort(result.begin(), result.begin() + K, result.end());\n\n std::vector res;\n for(unsigned int j = 0; j < K; j++) res.push_back(result[j].second);\n nn_results.push_back(res);\n\n }\n //std::cout<<\"bad query number: \" << VisitBucketNum[radius+1] << std::endl;\n }\n\n void getNeighbors64(size_t K, const Matrix& query){\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< tbflag(features_.get_rows(), false);\n\n nn_results.clear();\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n tbflag.reset();\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(unsigned int h=0; h < QueryCode64.size(); h++){\n unsigned long searchcode = QueryCode64[h][cur] ^ HammingBallMask64[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned long idx2 = searchcode - (( unsigned long)idx1 << lowbits);\n\n HashBucket64::iterator bucket= htb64[h][idx1].find(idx2);\n if(bucket != htb64[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n\n if (p> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n std::partial_sort(result.begin(), result.begin() + K, result.end());\n\n std::vector res;\n for(unsigned int j = 0; j < K; j++) res.push_back(result[j].second);\n nn_results.push_back(res);\n }\n //std::cout<<\"bad query number: \" <& query){\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< (unsigned)SP.extend_to)\n resultSize = K;\n\n boost::dynamic_bitset<> tbflag(features_.get_rows(), false);\n nn_results.clear();\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n tbflag.reset();\n\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(size_t h=0; h < QueryCode.size(); h++){\n unsigned int searchcode = QueryCode[h][cur] ^ HammingBallMask[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned int idx2 = searchcode - (idx1 << lowbits);\n\n HashBucket::iterator bucket= htb[h][idx1].find(idx2);\n if(bucket != htb[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n int base_n = features_.get_rows();\n while(p < (unsigned int)SP.search_init_num){\n unsigned int nn = rand() % base_n;\n if(tbflag.test(nn)) continue;\n tbflag.set(nn);\n pool[p++] = (nn);\n }\n\n //sorting the pool\n std::vector> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n std::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n result.resize(resultSize);\n pool.clear();\n for(int j = 0; j < resultSize; j++)\n pool.push_back(result[j].second);\n\n //nn_exp\n boost::dynamic_bitset<> newflag(features_.get_rows(), true);\n newflag.set();\n\n int iter=0;\n std::vector ids;\n while(iter++ < SP.search_epoches){\n //the heap is max heap\n ids.clear();\n for(unsigned j = 0; j < SP.extend_to ; j++){\n if(newflag.test( pool[j] )){\n newflag.reset(pool[j]);\n\n for(unsigned neighbor=0; neighbor < gs[pool[j]].size(); neighbor++){\n unsigned id = gs[pool[j]][neighbor];\n\n if(tbflag.test(id))continue;\n else tbflag.set(id);\n\n ids.push_back(id);\n }\n }\n }\n\n for(size_t j = 0; j < ids.size(); j++){\n result.push_back(std::make_pair(distance_->compare(query.get_row(cur), features_.get_row(ids[j]), features_.get_cols()),ids[j]));\n }\n std::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n result.resize(resultSize);\n pool.clear();\n for(int j = 0; j < resultSize; j++)\n pool.push_back(result[j].second);\n }\n\n if(K<(unsigned)SP.extend_to)\n pool.resize(K);\n\n nn_results.push_back(pool);\n }\n }\n\n void getNeighborsIEH32_kgraph(size_t K, const Matrix& query){\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< tbflag(features_.get_rows(), false);\n\n bool bSorted = true;\n unsigned pool_size = SP.search_epoches * SP.extend_to;\n if (pool_size >= (unsigned)SP.search_init_num){\n SP.search_init_num = pool_size;\n bSorted = false;\n }\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n\n tbflag.reset();\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(size_t h=0; h < QueryCode.size(); h++){\n unsigned int searchcode = QueryCode[h][cur] ^ HammingBallMask[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned int idx2 = searchcode - (idx1 << lowbits);\n\n HashBucket::iterator bucket= htb[h][idx1].find(idx2);\n if(bucket != htb[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n int base_n = features_.get_rows();\n while(p < (unsigned int)SP.search_init_num){\n unsigned int nn = rand() % base_n;\n if(tbflag.test(nn)) continue;\n tbflag.set(nn);\n pool[p++] = (nn);\n }\n\n std::vector> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n if(bSorted){\n std::partial_sort(result.begin(), result.begin() + pool_size, result.end());\n result.resize(pool_size);\n }\n\n\n tbflag.reset();\n\n std::vector knn(K + SP.extend_to +1);\n std::vector results;\n\n for (unsigned iter = 0; iter < (unsigned)SP.search_epoches; iter++) {\n\n unsigned L = 0;\n for(unsigned j=0; j < SP.extend_to ; j++){\n if(!tbflag.test(result[iter*SP.extend_to+j].second)){\n tbflag.set(result[iter*SP.extend_to+j].second);\n knn[L].id = result[iter*SP.extend_to+j].second;\n knn[L].dist = result[iter*SP.extend_to+j].first;\n knn[L].flag = true;\n L++;\n }\n }\n if(~bSorted){\n std::sort(knn.begin(), knn.begin() + L);\n }\n\n unsigned int k = 0;\n while (k < L) {\n unsigned int nk = L;\n if (knn[k].flag) {\n knn[k].flag = false;\n unsigned n = knn[k].id;\n\n for(unsigned neighbor=0; neighbor < gs[n].size(); neighbor++){\n unsigned id = gs[n][neighbor];\n\n if(tbflag.test(id))continue;\n tbflag.set(id);\n\n float dist = distance_->compare(query.get_row(cur), features_.get_row(id), features_.get_cols());\n Point nn(id, dist);\n unsigned int r = InsertIntoKnn(&knn[0], L, nn);\n\n //if ( (r <= L) && (L + 1 < knn.size())) ++L;\n if ( L + 1 < knn.size()) ++L;\n if (r < nk) nk = r;\n }\n }\n if (nk <= k) k = nk;\n else ++k;\n }\n\n\n if (L > K) L = K;\n if (results.empty()) {\n results.reserve(K + 1);\n results.resize(L + 1);\n std::copy(knn.begin(), knn.begin() + L, results.begin());\n }\n else {\n for (unsigned int l = 0; l < L; ++l) {\n unsigned r = InsertIntoKnn(&results[0], results.size() - 1, knn[l]);\n if (r < results.size() /* inserted */ && results.size() < (K + 1)) {\n results.resize(results.size() + 1);\n }\n }\n }\n }\n\n std::vector res;\n for(size_t i = 0; i < K && i < results.size();i++)\n res.push_back(results[i].id);\n nn_results.push_back(res);\n\n }\n }\n\n void getNeighborsIEH64_nnexp(size_t K, const Matrix& query){\n\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< (unsigned)SP.extend_to)\n resultSize = K;\n\n boost::dynamic_bitset<> tbflag(features_.get_rows(), false);\n nn_results.clear();\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n tbflag.reset();\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(unsigned int h=0; h < QueryCode64.size(); h++){\n unsigned long searchcode = QueryCode64[h][cur] ^ HammingBallMask64[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned long idx2 = searchcode - (( unsigned long)idx1 << lowbits);\n\n HashBucket64::iterator bucket= htb64[h][idx1].find(idx2);\n if(bucket != htb64[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n int base_n = features_.get_rows();\n while(p < (unsigned int)SP.search_init_num){\n unsigned int nn = rand() % base_n;\n if(tbflag.test(nn)) continue;\n tbflag.set(nn);\n pool[p++] = (nn);\n }\n\n //sorting the pool\n std::vector> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n std::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n result.resize(resultSize);\n pool.clear();\n for(int j = 0; j < resultSize; j++)\n pool.push_back(result[j].second);\n\n //nn_exp\n boost::dynamic_bitset<> newflag(features_.get_rows(), true);\n newflag.set();\n\n int iter=0;\n std::vector ids;\n while(iter++ < SP.search_epoches){\n //the heap is max heap\n ids.clear();\n for(unsigned j = 0; j < SP.extend_to ; j++){\n if(newflag.test( pool[j] )){\n newflag.reset(pool[j]);\n\n for(unsigned neighbor=0; neighbor < gs[pool[j]].size(); neighbor++){\n unsigned id = gs[pool[j]][neighbor];\n\n if(tbflag.test(id))continue;\n else tbflag.set(id);\n\n ids.push_back(id);\n }\n }\n }\n\n for(size_t j = 0; j < ids.size(); j++){\n result.push_back(std::make_pair(distance_->compare(query.get_row(cur), features_.get_row(ids[j]), features_.get_cols()),ids[j]));\n }\n std::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n result.resize(resultSize);\n pool.clear();\n for(int j = 0; j < resultSize; j++)\n pool.push_back(result[j].second);\n }\n\n if(K<(unsigned)SP.extend_to)\n pool.resize(K);\n\n nn_results.push_back(pool);\n }\n }\n\n void getNeighborsIEH64_kgraph(size_t K, const Matrix& query){\n int lowbits = codelength - upbits;\n\n unsigned int MaxCheck=HammingRadius[radius];\n std::cout<<\"maxcheck : \"< tbflag(features_.get_rows(), false);\n\n bool bSorted = true;\n unsigned pool_size = SP.search_epoches * SP.extend_to;\n if (pool_size >= (unsigned)SP.search_init_num){\n SP.search_init_num = pool_size;\n bSorted = false;\n }\n\n VisitBucketNum.clear();\n VisitBucketNum.resize(radius+2);\n\n\n for(size_t cur = 0; cur < query.get_rows(); cur++){\n\n std::vector pool(SP.search_init_num);\n unsigned int p = 0;\n tbflag.reset();\n\n unsigned int j = 0;\n for(; j < MaxCheck; j++){\n for(unsigned int h=0; h < QueryCode64.size(); h++){\n unsigned long searchcode = QueryCode64[h][cur] ^ HammingBallMask64[j];\n unsigned int idx1 = searchcode >> lowbits;\n unsigned long idx2 = searchcode - (( unsigned long)idx1 << lowbits);\n\n HashBucket64::iterator bucket= htb64[h][idx1].find(idx2);\n if(bucket != htb64[h][idx1].end()){\n std::vector vp = bucket->second;\n for(size_t k = 0; k < vp.size() && p < (unsigned int)SP.search_init_num; k++){\n if(tbflag.test(vp[k]))continue;\n\n tbflag.set(vp[k]);\n pool[p++]=(vp[k]);\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n if(p >= (unsigned int)SP.search_init_num) break;\n }\n\n if(p < (unsigned int)SP.search_init_num){\n VisitBucketNum[radius+1]++;\n }else{\n for(int r=0;r<=radius;r++){\n if(j<=HammingRadius[r]){\n VisitBucketNum[r]++;\n break;\n }\n }\n }\n\n int base_n = features_.get_rows();\n while(p < (unsigned int)SP.search_init_num){\n unsigned int nn = rand() % base_n;\n if(tbflag.test(nn)) continue;\n tbflag.set(nn);\n pool[p++] = (nn);\n }\n\n std::vector> result;\n for(unsigned int i=0; icompare(query.get_row(cur), features_.get_row(pool[i]), features_.get_cols()),pool[i]));\n }\n if(bSorted){\n std::partial_sort(result.begin(), result.begin() + pool_size, result.end());\n result.resize(pool_size);\n }\n\n\n tbflag.reset();\n\n std::vector knn(K + SP.extend_to +1);\n std::vector results;\n\n for (unsigned iter = 0; iter < (unsigned)SP.search_epoches; iter++) {\n\n unsigned L = 0;\n for(unsigned j=0; j < (unsigned)SP.extend_to ; j++){\n if(!tbflag.test(result[iter*SP.extend_to+j].second)){\n tbflag.set(result[iter*SP.extend_to+j].second);\n knn[L].id = result[iter*SP.extend_to+j].second;\n knn[L].dist = result[iter*SP.extend_to+j].first;\n knn[L].flag = true;\n L++;\n }\n }\n if(~bSorted){\n std::sort(knn.begin(), knn.begin() + L);\n }\n\n unsigned int k = 0;\n while (k < L) {\n unsigned int nk = L;\n if (knn[k].flag) {\n knn[k].flag = false;\n unsigned n = knn[k].id;\n\n for(unsigned neighbor=0; neighbor < gs[n].size(); neighbor++){\n unsigned id = gs[n][neighbor];\n\n if(tbflag.test(id))continue;\n tbflag.set(id);\n\n float dist = distance_->compare(query.get_row(cur), features_.get_row(id), features_.get_cols());\n Point nn(id, dist);\n unsigned int r = InsertIntoKnn(&knn[0], L, nn);\n\n //if ( (r <= L) && (L + 1 < knn.size())) ++L;\n if ( L + 1 < knn.size()) ++L;\n if (r < nk) nk = r;\n }\n }\n if (nk <= k) k = nk;\n else ++k;\n }\n\n\n if (L > K) L = K;\n if (results.empty()) {\n results.reserve(K + 1);\n results.resize(L + 1);\n std::copy(knn.begin(), knn.begin() + L, results.begin());\n }\n else {\n for (unsigned int l = 0; l < L; ++l) {\n unsigned r = InsertIntoKnn(&results[0], results.size() - 1, knn[l]);\n if (r < results.size() /* inserted */ && results.size() < (K + 1)) {\n results.resize(results.size() + 1);\n }\n }\n }\n }\n\n std::vector res;\n for(size_t i = 0; i < K && i < results.size();i++)\n res.push_back(results[i].id);\n nn_results.push_back(res);\n\n }\n }\n\n void outputVisitBucketNum(){\n unsigned i=0;\n std::cout<< \"Radius \" << i <<\" bucket num: \"< heap;\n in.read((char*)&dim, sizeof(int));\n for(int j =0; j < dim; j++){\n unsigned id;\n in.read((char*)&id, sizeof(int));\n heap.push_back(id);\n }\n gs.push_back(heap);\n }\n in.close();\n }\n void saveGraph(char* filename){}\n void initGraph(){}\n\n protected:\n int tablenum;\n int upbits;\n int codelength;\n int codelengthshift;\n int radius;\n USING_BASECLASS_SYMBOLS\n\n std::vector htb;\n std::vector BaseCode;\n std::vector QueryCode;\n std::vector HammingBallMask;\n\n std::vector htb64;\n std::vector BaseCode64;\n std::vector QueryCode64;\n std::vector HammingBallMask64;\n\n std::vector HammingRadius;\n\n // for statistic info\n std::vector VisitBucketNum;\n\n};\n}\n#endif\n" }, { "path": "algorithm/init_indices.hpp", "content": "#ifndef EFANNA_INIT_INDICES_H_\n#define EFANNA_INIT_INDICES_H_\n#include \"base_index.hpp\"\n#include \"kdtreeub_index.hpp\"\n#include \"hashing_index.hpp\"\nnamespace efanna{\n\n\n template class Index, typename DataType>\n inline InitIndex* create_index_(efanna::Matrix data, const efanna::IndexParams& params, const Distance* d)\n {\n return new Index(data, d, params);\n }\n\n template \n inline InitIndex*\n create_index_by_type(const init_algorithm index_type,\n \t\tconst Matrix& dataset, const IndexParams& params, const Distance* d)\n {\n InitIndex* initIndex = NULL;\n switch(index_type){\n case KDTREE_UB:\n initIndex = create_index_(dataset, params, d);\n break;\n case HASHING:\n initIndex = create_index_(dataset, params, d);\n break;\n }\n return initIndex;\n }\n}\n#endif\n" }, { "path": "algorithm/kdtreeub_index.hpp", "content": "#ifndef EFANNA_KDTREE_UB_INDEX_H_\n#define EFANNA_KDTREE_UB_INDEX_H_\n#include \"algorithm/base_index.hpp\"\n#include \n#include \n#include \n#include \n#include \n//#include \n//using std::bitset;\n#include \n\nnamespace efanna{\nstruct KDTreeUbIndexParams : public IndexParams\n{\n\tKDTreeUbIndexParams(bool rnn_used, int tree_num_total, int merge_level = 4, int epoches = 4, int check = 25, int myL = 30, int building_use_k = 10, int tree_num_build = 0, int myS = 10)\n\t{\n\t\treverse_nn_used = rnn_used;\n\t\tinit_index_type = KDTREE_UB;\n\t\tK = building_use_k;\n\t\tbuild_epoches = epoches;\n\t\tS = myS;\n\t\tValueType treev;\n\t\ttreev.int_val = tree_num_total;\n\t\textra_params.insert(std::make_pair(\"trees\",treev));\n\t\tValueType treeb;\n\t\ttreeb.int_val = tree_num_build > 0 ? tree_num_build : tree_num_total;\n\t\textra_params.insert(std::make_pair(\"treesb\",treeb));\n\t\tValueType merge_levelv;\n\t\tmerge_levelv.int_val = merge_level;\n\t\textra_params.insert(std::make_pair(\"ml\",merge_levelv));\n\t\tL = myL;\n\t\tCheck_K = check;\n\t}\n};\ntemplate \nclass KDTreeUbIndex : public InitIndex\n{\npublic:\n\n\ttypedef InitIndex BaseClass;\n\tKDTreeUbIndex(const Matrix& dataset, const Distance* d, const IndexParams& params = KDTreeUbIndexParams(true,4)) :\n\t\tBaseClass(dataset,d,params)\n\t{\n\t\tstd::cout<<\"kdtree ub initial\"<second).int_val;\n#ifdef INFO\n\t\t\tstd::cout << \"Using kdtree to build \"<< TreeNum << \" trees in total\" << std::endl;\n#endif\n\t\t}\n\t\telse{\n\t\t\tTreeNum = 4;\n#ifdef INFO\n\t\t\tstd::cout << \"Using kdtree to build \"<< TreeNum << \" trees in total\" << std::endl;\n#endif\n\t\t}\n\t\tSP.tree_num = TreeNum;\n\n\t\tit = params_.extra_params.find(\"treesb\");\n\t\tif(it != params_.extra_params.end()){\n\t\t\tTreeNumBuild = (it->second).int_val;\n#ifdef INFO\n\t\t\tstd::cout << \"Building kdtree graph with \"<< TreeNumBuild <<\" trees\"<< std::endl;\n#endif\n\t\t}\n\t\telse{\n\t\t\tTreeNumBuild = TreeNum;\n#ifdef INFO\n\t\t\tstd::cout << \"Building kdtree graph with \"<< TreeNumBuild <<\" trees\"<< std::endl;\n#endif\n\t\t}\n\n\t\tit = params_.extra_params.find(\"ml\");\n\t\tif(it != params_.extra_params.end()){\n\t\t\tml = (it->second).int_val;\n#ifdef INFO\n\t\t\tstd::cout << \"Building kdtree initial index with merge level \"<< ml << std::endl;\n#endif\n\t\t}\n\t\telse{\n\t\t\tml = -1;\n#ifdef INFO\n\t\t\tstd::cout << \"Building kdtree initial index with max merge level \"<< std::endl;\n#endif\n\t\t}\n\t\tmax_deepth = 0x0fffffff;\n\t\terror_flag = false;\n\t}\n\n\tvoid buildIndexImpl(){\n#ifdef INFO\n\t\tclock_t s,f;\n\t\ts = clock();\n#endif\n\t\tinitGraph();\n\n#ifdef INFO\n\t\tf = clock();\n#endif\n\n\t\tstd::cout << \"initial graph finised\"<< std::endl;\n#ifdef INFO\n\t\tstd::cout << \"initial graph using time: \"<< (f-s)*1.0/CLOCKS_PER_SEC<<\" seconds\"<< std::endl;\n#endif\n\n\t\tif(error_flag){\n\t\t\tstd::cout << \"merge level deeper than tree, max merge deepth is\" << max_deepth-1<~Node();\n\t\t\tif (Rchild!=NULL) Rchild->~Node();\n\t\t}\n\n\t};\n\n\tvoid loadIndex(char* filename){\n\t\tread_data(filename);\n\t}\n\tvoid saveIndex(char* filename){\n\n\t\tsize_t points_num = features_.get_rows();\n\t\tsize_t feature_dim = features_.get_cols();\n\t\tsave_data(filename, params_.K, points_num, feature_dim);\n\t}\n\t//algorithms copy and rewrite from flann\n\tvoid loadTrees(char* filename){\n\t\tstd::ifstream in(filename, std::ios::binary|std::ios::in);\n\t\tif(!in.is_open()){std::cout<<\"open file error\"< tree_nodes;\n\t\t\tfor(int j=0;jDivDim),sizeof(tmp->DivDim));\n\t\t\t\tin.read((char*)&(tmp->DivVal),sizeof(tmp->DivVal));\n\t\t\t\tin.read((char*)&(tmp->StartIdx),sizeof(tmp->StartIdx));\n\t\t\t\tin.read((char*)&(tmp->EndIdx),sizeof(tmp->EndIdx));\n\t\t\t\tin.read((char*)&(tmp->Lchild),sizeof(tmp->Lchild));\n\t\t\t\tin.read((char*)&(tmp->Rchild),sizeof(tmp->Rchild));\n\t\t\t\ttmp->Lchild = NULL;\n\t\t\t\ttmp->Rchild = NULL;\n\t\t\t\ttmp->treeid = i;\n\t\t\t\ttree_nodes.push_back(tmp);\n\n\n\t\t\t}\n\t\t\t//std::cout<<\"build \"< leaves;\n\t\t\tfor(unsigned int j=0;j::iterator it;//int cnt=0;\n\t\tfor(it=tree_roots_.begin(); it!=tree_roots_.end(); it++){\n\t\t\t//write tree nodes with depth first trace\n\n\n\t\t\tsize_t offset_node_num = out.tellp();\n\n\t\t\tout.seekp(sizeof(int),std::ios::cur);\n\n\t\t\tunsigned int node_size = sizeof(struct Node);\n\t\t\tout.write((char *)&(node_size), sizeof(int));\n\n\t\t\tunsigned int node_num = DepthFirstWrite(out, *it);\n\n\t\t\tout.seekg(offset_node_num,std::ios::beg);\n\n\t\t\tout.write((char *)&(node_num), sizeof(int));\n\n\t\t\tout.seekp(0,std::ios::end);\n\t\t\t//std::cout<<\"tree: \"<row_id;\n out.write((char*)&id, sizeof(int));\n }\n }//meansize /= knn_graph.size();\n //std::cout << \"size mean \" << meansize << std::endl;\n out.close();\n }\n\t */\n\tvoid saveGraph(char* filename){\n\t\tstd::ofstream out(filename,std::ios::binary);\n\t\tunsigned N = gs.size();\n\t\t//out.write((char*)&N, sizeof(int));\n\t\tfor(unsigned i=0; i < N; i++){\n\t\t\tunsigned k = gs[i].size();\n\t\t\t//unsigned m = M[i];\n\t\t\t//DataType norm = norms[i];\n\t\t\tout.write((char*)&k, sizeof(unsigned));\n\t\t\t//out.write((char*)&m, sizeof(unsigned));\n\t\t\t//out.write((char*)&norm, sizeof(DataType));\n\t\t\tfor(unsigned j = 0; j < k; j++){\n\t\t\t\tunsigned id = gs[i][j];\n\t\t\t\tout.write((char*)&id, sizeof(unsigned));\n\t\t\t}\n\t\t}\n\t\tout.close();\n\t}\n\t//for nn search\n\n\tvoid SearchQueryToLeaf(Node* node, const DataType* q, unsigned dep, std::vector& node_pool){\n\t\tif(node->Lchild != NULL && node->Rchild !=NULL){\n\t\t\tif(q[node->DivDim] < node->DivVal){\n\t\t\t\tSearchQueryToLeaf(node->Lchild, q, dep, node_pool);\n\t\t\t\tif(node_pool.size() < dep)\n\t\t\t\t\tSearchQueryToLeaf(node->Rchild, q, dep, node_pool);\n\t\t\t}\n\t\t\telse{\n\t\t\t\tSearchQueryToLeaf(node->Rchild, q, dep, node_pool);\n\t\t\t\tif(node_pool.size() < dep)\n\t\t\t\t\tSearchQueryToLeaf(node->Lchild, q, dep, node_pool);\n\t\t\t}\n\t\t}\n\t\telse\n\t\t\tnode_pool.push_back(node);\n\t}\n\n\tvoid getSearchNodeList(Node* node, const DataType* q, unsigned int lsize, std::vector& vn){\n\t\tif(vn.size() >= lsize)\n\t\t\treturn;\n\n\t\tif(node->Lchild != NULL && node->Rchild !=NULL){\n\t\t\tif(q[node->DivDim] < node->DivVal){\n\t\t\t\tgetSearchNodeList(node->Lchild, q, lsize, vn );\n\t\t\t\tgetSearchNodeList(node->Rchild, q, lsize, vn);\n\t\t\t}else{\n\t\t\t\tgetSearchNodeList(node->Rchild, q, lsize, vn);\n\t\t\t\tgetSearchNodeList(node->Lchild, q, lsize, vn);\n\t\t\t}\n\t\t}else\n\t\t\tvn.push_back(node);\n\t}\n\n\n\tvoid getNeighbors(size_t searchK, const Matrix& query){\n\t\tswitch(SP.search_method){\n\t\tcase 0:\n\t\t\tgetNeighbors_nnexp(searchK, query);\n\t\t\tbreak;\n\t\tcase 1:\n\t\t\tgetNeighbors_kgraph(searchK, query);\n\t\t\tbreak;\n\t\tdefault:\n\t\t\tstd::cout<<\"no such searching method\"<& query){\n#ifdef INFO\n\t\tstd::cout<<\"using tree num \"<< SP.tree_num< tree_roots_.size()){\n\t\t\tstd::cout<<\"wrong tree number\"< (unsigned)SP.extend_to)\n\t\t\tresultSize = K;\n\n\n#pragma omp parallel for\n\t\tfor(unsigned int cur = 0; cur < query.get_rows(); cur++){\n\t\t\tboost::dynamic_bitset<> tbflag(features_.get_rows(), false);\n\t\t\tboost::dynamic_bitset<> newflag(features_.get_rows(), true);\n\t\t\ttbflag.reset();\n\t\t\tnewflag.set();\n\n\t\t\tstd::vector> NodeCandi;\n\t\t\tNodeCandi.resize(SP.tree_num);\n\n\t\t\tconst DataType* q_row = query.get_row(cur);\n\t\t\t_mm_prefetch((char *)q_row, _MM_HINT_T0);\n\t\t\tunsigned int lsize = SP.search_init_num*2 / (5*SP.tree_num) + 1;\n\t\t\tfor(unsigned int i = 0; i < SP.tree_num; i++){\n\t\t\t\tgetSearchNodeList(tree_roots_[i], q_row, lsize, NodeCandi[i]);\n\t\t\t}\n\t\t\tstd::vector pool(SP.search_init_num);\n\t\t\tunsigned int p = 0;\n\t\t\tfor(unsigned int ni = 0; ni < lsize; ni++){\n\t\t\t\tfor(unsigned int i = 0; i < NodeCandi.size(); i++){\n\t\t\t\t\tNode* leafn = NodeCandi[i][ni];\n\t\t\t\t\tfor(size_t j = leafn->StartIdx; j < leafn->EndIdx && p < (unsigned int)SP.search_init_num; j++){\n\t\t\t\t\t\tsize_t nn = LeafLists[i][j];\n\t\t\t\t\t\tif(tbflag.test(nn))continue;\n\t\t\t\t\t\ttbflag.set(nn);\n\t\t\t\t\t\tpool[p++]=(nn);\n\t\t\t\t\t}\n\t\t\t\t\tif(p >= (unsigned int)SP.search_init_num) break;\n\t\t\t\t}\n\t\t\t\tif(p >= (unsigned int)SP.search_init_num) break;\n\t\t\t}\n\t\t\tint base_n = features_.get_rows();\n\t\t\twhile(p < (unsigned int)SP.search_init_num){\n\t\t\t\tunsigned int nn = rand() % base_n;\n\t\t\t\tif(tbflag.test(nn))continue;\n\t\t\t\ttbflag.set(nn);\n\t\t\t\tpool[p++]=(nn);\n\t\t\t}\n\n\n\t\t\tstd::vector> result;\n\t\t\t//for(unsigned int i=0; i pool.size() ? pool.size() : s+cache_blocksz;\n\t\t\t\tunsigned s_ = s;\n\t\t\t\twhile(scompare(q_row, features_.get_row(pool[s_]), dim),pool[s_]));\n\t\t\t\t\ts_++;\n\t\t\t\t}\n\t\t\t}\n\t\t\tstd::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n\t\t\tresult.resize(resultSize);\n\t\t\tpool.clear();\n\t\t\tfor(int j = 0; j < resultSize; j++)\n\t\t\t\tpool.push_back(result[j].second);\n\n\t\t\tint iter=0;\n\t\t\tstd::vector ids;\n\t\t\twhile(iter++ < SP.search_epoches){\n\t\t\t\tids.clear();\n\t\t\t\tfor(unsigned j = 0; j < SP.extend_to ; j++){\n\t\t\t\t\tif(newflag.test( pool[j] )){\n\t\t\t\t\t\tnewflag.reset(pool[j]);\n\n\t\t\t\t\t\tfor(unsigned neighbor=0; neighbor < gs[pool[j]].size(); neighbor++){\n\t\t\t\t\t\t\tunsigned id = gs[pool[j]][neighbor];\n\n\t\t\t\t\t\t\tif(tbflag.test(id))continue;\n\t\t\t\t\t\t\telse tbflag.set(id);\n\n\t\t\t\t\t\t\tids.push_back(id);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t//for(unsigned int j=0; j ids.size() ? ids.size() : s+cache_blocksz;\n\t\t\t\t\tunsigned s_ = s;\n\t\t\t\t\twhile(scompare(q_row, features_.get_row(ids[s_]), dim),ids[s_]));\n\t\t\t\t\t\ts_++;\n\t\t\t\t\t}\n\t\t\t\t\t//result.push_back(std::make_pair(distance_->compare(q_row, features_.get_row(ids[j]), dim),ids[j]));\n\t\t\t\t}\n\t\t\t\tstd::partial_sort(result.begin(), result.begin() + resultSize, result.end());\n\t\t\t\tresult.resize(resultSize);\n\t\t\t\tpool.clear();\n\t\t\t\tfor(int j = 0; j < resultSize; j++)\n\t\t\t\t\tpool.push_back(result[j].second);\n\t\t\t}\n\n\t\t\tif(K& res = nn_results[cur];\n\t\t\tfor(unsigned i = 0; i < K ;i++)\n\t\t\t\tres.push_back(pool[i]);\n\t\t}\n\t}\n\n\tvoid getNeighbors_kgraph(size_t searchK, const Matrix& query){\n#ifdef INFO\n\t\tstd::cout<<\"using tree num \"<< SP.tree_num< tree_roots_.size()){\n\t\t\tstd::cout<<\"wrong tree number\"<= (unsigned)SP.search_init_num){\n\t\t\tSP.search_init_num = pool_size;\n\t\t\tbSorted = false;\n\t\t}\n\n#pragma omp parallel for\n\t\tfor(unsigned int cur = 0; cur < query.get_rows(); cur++){\n\t\t\tstd::mt19937 rng(1998);\n\t\t\tboost::dynamic_bitset<> flags(features_.get_rows(), false);\n\n\t\t\tstd::vector > Vnl;\n\t\t\tVnl.resize(SP.tree_num);\n\t\t\tconst DataType* q_row = query.get_row(cur);\n\t\t\t_mm_prefetch((char *)q_row, _MM_HINT_T0);\n\t\t\tfor(unsigned int i = 0; i < SP.tree_num; i++){\n\t\t\t\tgetSearchNodeList(tree_roots_[i], q_row, lsize, Vnl[i]);\n\t\t\t}\n\n\t\t\tstd::vector pool(SP.search_init_num);\n\t\t\tunsigned int p = 0;\n\t\t\tfor(unsigned int ni = 0; ni < lsize; ni++){\n\t\t\t\tfor(unsigned int i = 0; i < Vnl.size(); i++){\n\t\t\t\t\tNode* leafn = Vnl[i][ni];\n\t\t\t\t\tfor(size_t j = leafn->StartIdx; j < leafn->EndIdx && p < (unsigned int)SP.search_init_num; j++){\n\t\t\t\t\t\tsize_t nn = LeafLists[i][j];\n\t\t\t\t\t\tif(flags.test(nn))continue;\n\t\t\t\t\t\tflags.set(nn);\n\t\t\t\t\t\tpool[p++]=(nn);\n\t\t\t\t\t}\n\t\t\t\t\tif(p >= (unsigned int)SP.search_init_num) break;\n\t\t\t\t}\n\t\t\t\tif(p >= (unsigned int)SP.search_init_num) break;\n\t\t\t}\n\t\t\tint base_n = features_.get_rows();\n\t\t\twhile(p < (unsigned int)SP.search_init_num){\n\t\t\t\tunsigned int nn = rand() % base_n;\n\t\t\t\tif(flags.test(nn))continue;\n\t\t\t\tflags.set(nn);\n\t\t\t\tpool[p++]=(nn);\n\t\t\t}\n\n\t\t\tstd::vector> result;\n\t\t\tunsigned cache_blocksz = 80;\n\t\t\tfor(unsigned int i=0; i*cache_blocksz pool.size() ? pool.size() : s+cache_blocksz;\n\t\t\t\tunsigned s_ = s;\n\t\t\t\twhile(scompare(q_row, features_.get_row(pool[s_]), dim),pool[s_]));\n\t\t\t\t\ts_++;\n\t\t\t\t}\n\t\t\t}\n\t\t\tif(bSorted){\n\t\t\t\tstd::partial_sort(result.begin(), result.begin() + pool_size, result.end());\n\t\t\t\tresult.resize(pool_size);\n\t\t\t}\n\n\t\t\tflags.reset();\n\t\t\tstd::vector knn(searchK + SP.extend_to +1);\n\t\t\tstd::vector results;\n\t\t\tfor (unsigned iter = 0; iter < (unsigned)SP.search_epoches; ++iter) {\n\n\t\t\t\tunsigned L = 0;\n\t\t\t\tfor(unsigned j=0; j < (unsigned)SP.extend_to ; j++){\n\t\t\t\t\tif(!flags.test(result[iter*SP.extend_to+j].second)){\n\t\t\t\t\t\tflags.set(result[iter*SP.extend_to+j].second);\n\t\t\t\t\t\tknn[L].id = result[iter*SP.extend_to+j].second;\n\t\t\t\t\t\tknn[L].dist = result[iter*SP.extend_to+j].first;\n\t\t\t\t\t\tknn[L].flag = true;\n\t\t\t\t\t\tL++;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif(~bSorted){\n\t\t\t\t\tstd::sort(knn.begin(), knn.begin() + L);\n\t\t\t\t}\n\n\t\t\t\tunsigned k = 0;\n\t\t\t\twhile (k < L) {\n\t\t\t\t\tunsigned nk = L;\n\t\t\t\t\tif (knn[k].flag) {\n\t\t\t\t\t\tknn[k].flag = false;\n\t\t\t\t\t\tunsigned n = knn[k].id;\n\n\t\t\t\t\t\t//unsigned maxM = M[n];\n\t\t\t\t\t\tunsigned maxM = SP.extend_to;\n\t\t\t\t\t\t//if ((unsigned)SP.extend_to > maxM) maxM = SP.extend_to;\n\t\t\t\t\t\tauto const &neighbors = gs[n];\n\t\t\t\t\t\tif (maxM > neighbors.size()) {\n\t\t\t\t\t\t\tmaxM = neighbors.size();\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\tfor(unsigned m = 0; m < maxM; ++m){\n\t\t\t\t\t\t\t_mm_prefetch((char *)features_.get_row(neighbors[m]), _MM_HINT_T0);\n\t\t\t\t\t\t}\n\t\t\t\t\t\tfor (unsigned m = 0; m < maxM; ++m) {\n\t\t\t\t\t\t\tunsigned id = neighbors[m];\n\t\t\t\t\t\t\t//BOOST_VERIFY(id < graph.size());\n\t\t\t\t\t\t\tif (flags[id]) continue;\n\t\t\t\t\t\t\tflags[id] = true;\n\n\t\t\t\t\t\t\tDataType dist = distance_->compare(q_row, features_.get_row(id), dim);\n\n\t\t\t\t\t\t\tPoint nn(id, dist);\n\t\t\t\t\t\t\tunsigned r = InsertIntoKnn(&knn[0], L, nn);\n\t\t\t\t\t\t\t//BOOST_VERIFY(r <= L);\n\t\t\t\t\t\t\t//if (r > L) continue;\n\t\t\t\t\t\t\tif (L + 1 < knn.size()) ++L;\n\t\t\t\t\t\t\tif (r < nk) {\n\t\t\t\t\t\t\t\tnk = r;\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tif (nk <= k) {\n\t\t\t\t\t\tk = nk;\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\t++k;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif (L > searchK) L = searchK;\n\n\t\t\t\tif (results.empty()) {\n\t\t\t\t\tresults.reserve(searchK + 1);\n\t\t\t\t\tresults.resize(L + 1);\n\t\t\t\t\tstd::copy(knn.begin(), knn.begin() + L, results.begin());\n\t\t\t\t} else {\n\t\t\t\t\tfor (unsigned int l = 0; l < L; ++l) {\n\t\t\t\t\t\tunsigned r = InsertIntoKnn(&results[0], results.size() - 1, knn[l]);\n\t\t\t\t\t\tif (r < results.size() && results.size() < (searchK + 1)) {\n\t\t\t\t\t\t\tresults.resize(results.size() + 1);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tstd::vector& res = nn_results[cur];\n\t\t\tfor(size_t i = 0; i < searchK && i < results.size();i++)\n\t\t\t\tres.push_back(results[i].id);\n\t\t}\n\t}\n\n\n\n\tint DepthFirstWrite(std::fstream& out, struct Node *root){\n\t\tif(root==NULL) return 0;\n\t\tint left_cnt = DepthFirstWrite(out, root->Lchild);\n\t\tint right_cnt = DepthFirstWrite(out, root->Rchild);\n\n\t\t//std::cout << root->StartIdx <<\":\" << root->EndIdx<< std::endl;\n\t\tout.write((char *)&(root->DivDim), sizeof(root->DivDim));\n\t\tout.write((char *)&(root->DivVal), sizeof(root->DivVal));\n\t\tout.write((char *)&(root->StartIdx), sizeof(root->StartIdx));\n\t\tout.write((char *)&(root->EndIdx), sizeof(root->EndIdx));\n\t\tout.write((char *)&(root->Lchild), sizeof(root->Lchild));\n\t\tout.write((char *)&(root->Rchild), sizeof(root->Rchild));\n\t\treturn (left_cnt + right_cnt + 1);\n\t}\n\tstruct Node* DepthFirstBuildTree(std::vector& tree_nodes){\n\t\tstd::vector root_serial;\n\t\ttypename std::vector::iterator it = tree_nodes.begin();\n\t\tfor( ; it!=tree_nodes.end(); it++){\n\t\t\tNode* tmp = *it;\n\t\t\tsize_t rsize = root_serial.size();\n\t\t\tif(rsize<2){\n\t\t\t\troot_serial.push_back(tmp);\n\t\t\t\t//continue;\n\t\t\t}\n\t\t\telse{\n\t\t\t\tNode *last1 = root_serial[rsize-1];\n\t\t\t\tNode *last2 = root_serial[rsize-2];\n\t\t\t\tif(last1->EndIdx == tmp->EndIdx && last2->StartIdx == tmp->StartIdx){\n\t\t\t\t\ttmp->Rchild = last1;\n\t\t\t\t\ttmp->Lchild = last2;\n\t\t\t\t\troot_serial.pop_back();\n\t\t\t\t\troot_serial.pop_back();\n\t\t\t\t}\n\t\t\t\troot_serial.push_back(tmp);\n\t\t\t}\n\n\t\t}\n\t\tif(root_serial.size()!=1){\n\t\t\tstd::cout << \"Error constructing trees\" << std::endl;\n\t\t\treturn NULL;\n\t\t}\n\t\treturn root_serial[0];\n\t}\n\tvoid read_data(char *filename){\n\t\tstd::ifstream in(filename, std::ios::binary|std::ios::in);\n\t\tif(!in.is_open()){std::cout<<\"open file error\"< tree_nodes;\n\t\t\tfor(int j=0;jDivDim),sizeof(tmp->DivDim));\n\t\t\t\tin.read((char*)&(tmp->DivVal),sizeof(tmp->DivVal));\n\t\t\t\tin.read((char*)&(tmp->StartIdx),sizeof(tmp->StartIdx));\n\t\t\t\tin.read((char*)&(tmp->EndIdx),sizeof(tmp->EndIdx));\n\t\t\t\tin.read((char*)&(tmp->Lchild),sizeof(tmp->Lchild));\n\t\t\t\tin.read((char*)&(tmp->Rchild),sizeof(tmp->Rchild));\n\t\t\t\ttmp->Lchild = NULL;\n\t\t\t\ttmp->Rchild = NULL;\n\t\t\t\ttree_nodes.push_back(tmp);\n\n\n\t\t\t}\n\t\t\t//std::cout<<\"build \"< leaves;\n\t\t\tfor(unsigned int j=0;j can(id, -1);\n\t\t\t\theap.insert(can);\n\t\t\t}\n\t\t\tknn_graph.push_back(heap);\n\t\t}\n\t\tin.close();\n\t}\n\tvoid save_data(char *filename, unsigned int K, size_t num, size_t dim){\n\t\tstd::fstream out(filename, std::ios::binary|std::ios::out);\n\t\tif(!out.is_open()){std::cout<<\"open file error\"<::iterator it;//int cnt=0;\n\t\tfor(it=tree_roots_.begin(); it!=tree_roots_.end(); it++){\n\t\t\t//write tree nodes with depth first trace\n\n\n\t\t\tsize_t offset_node_num = out.tellp();\n\n\t\t\tout.seekp(sizeof(int),std::ios::cur);\n\n\t\t\tunsigned int node_size = sizeof(struct Node);\n\t\t\tout.write((char *)&(node_size), sizeof(int));\n\n\t\t\tunsigned int node_num = DepthFirstWrite(out, *it);\n\n\t\t\tout.seekg(offset_node_num,std::ios::beg);\n\n\t\t\tout.write((char *)&(node_num), sizeof(int));\n\n\t\t\tout.seekp(0,std::ios::end);\n\t\t\t//std::cout<<\"tree: \"<row_id;\n\t\t\t\tout.write((char*)&id, sizeof(int));\n\t\t\t}\n\t\t}\n\n\t\tout.close();\n\t}\n\t/*\n Node* divideTree(std::mt19937& rng, int* indices, size_t count, size_t offset){\n Node* node = new Node();\n if(count <= params_.TNS){\n node->DivDim = -1;\n node->Lchild = NULL;\n node->Rchild = NULL;\n node->StartIdx = offset;\n node->EndIdx = offset + count;\n //add points\n\n for(size_t i = 0; i < count; i++){\n for(size_t j = i+1; j < count; j++){\n DataType dist = distance_->compare(\n features_.get_row(indices[i]), features_.get_row(indices[j]), features_.get_cols());\n\n if(knn_graph[indices[i]].size() < params_.S || dist < knn_graph[indices[i]].begin()->distance){\n Candidate c1(indices[j], dist);\n knn_graph[indices[i]].insert(c1);\n if(knn_graph[indices[i]].size() > params_.S)knn_graph[indices[i]].erase(knn_graph[indices[i]].begin());\n }\n else if(nhoods[indices[i]].nn_new.size() < params_.S * 2)nhoods[indices[i]].nn_new.push_back(indices[j]);\n if(knn_graph[indices[j]].size() < params_.S || dist < knn_graph[indices[j]].begin()->distance){\n Candidate c2(indices[i], dist);\n knn_graph[indices[j]].insert(c2);\n if(knn_graph[indices[j]].size() > params_.S)knn_graph[indices[j]].erase(knn_graph[indices[j]].begin());\n }\n else if(nhoods[indices[j]].nn_new.size() < params_.S * 2)nhoods[indices[j]].nn_new.push_back(indices[i]);\n }\n }\n\n }else{\n int idx;\n int cutdim;\n DataType cutval;\n meanSplit(rng, indices, count, idx, cutdim, cutval);\n\n node->DivDim = cutdim;\n node->DivVal = cutval;\n node->StartIdx = offset;\n node->EndIdx = offset + count;\n node->Lchild = divideTree(rng, indices, idx, offset);\n node->Rchild = divideTree(rng, indices+idx, count-idx, offset+idx);\n }\n\n return node;\n }\n\n Node* divideTreeOnly(std::mt19937& rng, unsigned* indices, size_t count, size_t offset){\n Node* node = new Node();\n if(count <= params_.TNS){\n node->DivDim = -1;\n node->Lchild = NULL;\n node->Rchild = NULL;\n node->StartIdx = offset;\n node->EndIdx = offset + count;\n //add points\n\n }else{\n unsigned idx;\n unsigned cutdim;\n DataType cutval;\n meanSplit(rng, indices, count, idx, cutdim, cutval);\n\n node->DivDim = cutdim;\n node->DivVal = cutval;\n node->StartIdx = offset;\n node->EndIdx = offset + count;\n node->Lchild = divideTreeOnly(rng, indices, idx, offset);\n node->Rchild = divideTreeOnly(rng, indices+idx, count-idx, offset+idx);\n }\n\n return node;\n }\n\t */\n\n\tvoid meanSplit(std::mt19937& rng, unsigned* indices, unsigned count, unsigned& index, unsigned& cutdim, DataType& cutval){\n\t\tsize_t veclen_ = features_.get_cols();\n\t\tDataType* mean_ = new DataType[veclen_];\n\t\tDataType* var_ = new DataType[veclen_];\n\t\tmemset(mean_,0,veclen_*sizeof(DataType));\n\t\tmemset(var_,0,veclen_*sizeof(DataType));\n\n\t\t/* Compute mean values. Only the first SAMPLE_NUM values need to be\n sampled to get a good estimate.\n\t\t */\n\t\tunsigned cnt = std::min((unsigned)SAMPLE_NUM+1, count);\n\t\tfor (unsigned j = 0; j < cnt; ++j) {\n\t\t\tconst DataType* v = features_.get_row(indices[j]);\n\t\t\tfor (size_t k=0; kcount/2) index = lim1;\n\t\telse if (lim2=cutval) --right;\n\t\t\tif (left>right) break;\n\t\t\tstd::swap(indices[left], indices[right]); ++left; --right;\n\t\t}\n\t\tlim1 = left;//lim1 is the id of the leftmost point <= cutval\n\t\tright = count-1;\n\t\tfor (;; ) {\n\t\t\twhile (left<=right && features_.get_row(indices[left])[cutdim]<=cutval) ++left;\n\t\t\twhile (left<=right && features_.get_row(indices[right])[cutdim]>cutval) --right;\n\t\t\tif (left>right) break;\n\t\t\tstd::swap(indices[left], indices[right]); ++left; --right;\n\t\t}\n\t\tlim2 = left;//lim2 is the id of the leftmost point >cutval\n\t}\n\tint selectDivision(std::mt19937& rng, DataType* v){\n\t\tint num = 0;\n\t\tsize_t topind[RAND_DIM];\n\n\t\t//Create a list of the indices of the top RAND_DIM values.\n\t\tfor (size_t i = 0; i < features_.get_cols(); ++i) {\n\t\t\tif ((num < RAND_DIM)||(v[i] > v[topind[num-1]])) {\n\t\t\t\t// Put this element at end of topind.\n\t\t\t\tif (num < RAND_DIM) {\n\t\t\t\t\ttopind[num++] = i; // Add to list.\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\ttopind[num-1] = i; // Replace last element.\n\t\t\t\t}\n\t\t\t\t// Bubble end value down to right location by repeated swapping. sort the varience in decrease order\n\t\t\t\tint j = num - 1;\n\t\t\t\twhile (j > 0 && v[topind[j]] > v[topind[j-1]]) {\n\t\t\t\t\tstd::swap(topind[j], topind[j-1]);\n\t\t\t\t\t--j;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t// Select a random integer in range [0,num-1], and return that index.\n\t\tint rnd = rng()%num;\n\t\treturn (int)topind[rnd];\n\t}\n\tvoid getMergeLevelNodeList(Node* node, size_t treeid, int deepth){\n\t\tif(node->Lchild != NULL && node->Rchild != NULL && deepth < ml){\n\t\t\tdeepth++;\n\t\t\tgetMergeLevelNodeList(node->Lchild, treeid, deepth);\n\t\t\tgetMergeLevelNodeList(node->Rchild, treeid, deepth);\n\t\t}else if(deepth == ml){\n\t\t\tmlNodeList.push_back(std::make_pair(node,treeid));\n\t\t}else{\n\t\t\terror_flag = true;\n\t\t\tif(deepth < max_deepth)max_deepth = deepth;\n\t\t}\n\t}\n\tNode* SearchToLeaf(Node* node, size_t id){\n\t\tif(node->Lchild != NULL && node->Rchild !=NULL){\n\t\t\tif(features_.get_row(id)[node->DivDim] < node->DivVal)\n\t\t\t\treturn SearchToLeaf(node->Lchild, id);\n\t\t\telse\n\t\t\t\treturn SearchToLeaf(node->Rchild, id);\n\t\t}\n\t\telse\n\t\t\treturn node;\n\t}int cc = 0;\n\tvoid mergeSubGraphs(size_t treeid, Node* node){\n\t\tif(node->Lchild != NULL && node->Rchild != NULL){\n\t\t\tmergeSubGraphs(treeid, node->Lchild);\n\t\t\tmergeSubGraphs(treeid, node->Rchild);\n\n\t\t\tsize_t numL = node->Lchild->EndIdx - node->Lchild->StartIdx;\n\t\t\tsize_t numR = node->Rchild->EndIdx - node->Rchild->StartIdx;\n\t\t\tsize_t start,end;\n\t\t\tNode * root;\n\t\t\tif(numL < numR){\n\t\t\t\troot = node->Rchild;\n\t\t\t\tstart = node->Lchild->StartIdx;\n\t\t\t\tend = node->Lchild->EndIdx;\n\t\t\t}else{\n\t\t\t\troot = node->Lchild;\n\t\t\t\tstart = node->Rchild->StartIdx;\n\t\t\t\tend = node->Rchild->EndIdx;\n\t\t\t}\n\n\t\t\tfor(;start < end; start++){\n\n\t\t\t\tsize_t feature_id = LeafLists[treeid][start];\n\n\t\t\t\tNode* leaf = SearchToLeaf(root, feature_id);\n\t\t\t\tfor(size_t i = leaf->StartIdx; i < leaf->EndIdx; i++){\n\t\t\t\t\tsize_t tmpfea = LeafLists[treeid][i];\n\t\t\t\t\tDataType dist = distance_->compare(\n\t\t\t\t\t\t\tfeatures_.get_row(tmpfea), features_.get_row(feature_id), features_.get_cols());\n\n\t\t\t\t\t{LockGuard g(*nhoods[tmpfea].lock);\n\t\t\t\t\tif(knn_graph[tmpfea].size() < params_.S || dist < knn_graph[tmpfea].begin()->distance){\n\t\t\t\t\t\tCandidate c1(feature_id, dist);\n\t\t\t\t\t\tknn_graph[tmpfea].insert(c1);\n\t\t\t\t\t\tif(knn_graph[tmpfea].size() > params_.S)knn_graph[tmpfea].erase(knn_graph[tmpfea].begin());\n\n\n\t\t\t\t\t}\n\t\t\t\t\telse if(nhoods[tmpfea].nn_new.size() < params_.S * 2){\n\n\t\t\t\t\t\tnhoods[tmpfea].nn_new.push_back(feature_id);\n\n\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t{LockGuard g(*nhoods[feature_id].lock);\n\t\t\t\t\tif(knn_graph[feature_id].size() < params_.S || dist < knn_graph[feature_id].begin()->distance){\n\t\t\t\t\t\tCandidate c1(tmpfea, dist);\n\t\t\t\t\t\tknn_graph[feature_id].insert(c1);\n\t\t\t\t\t\tif(knn_graph[feature_id].size() > params_.S)knn_graph[feature_id].erase(knn_graph[feature_id].begin());\n\n\t\t\t\t\t}\n\t\t\t\t\telse if(nhoods[feature_id].nn_new.size() < params_.S * 2){\n\n\t\t\t\t\t\tnhoods[feature_id].nn_new.push_back(tmpfea);\n\n\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\ttypedef std::set, std::greater> > CandidateHeap;\n\n\nprotected:\n\tenum\n\t{\n\t\t/**\n\t\t * To improve efficiency, only SAMPLE_NUM random values are used to\n\t\t * compute the mean and variance at each level when building a tree.\n\t\t * A value of 100 seems to perform as well as using all values.\n\t\t */\n\t\tSAMPLE_NUM = 100,\n\t\t/**\n\t\t * Top random dimensions to consider\n\t\t *\n\t\t * When creating random trees, the dimension on which to subdivide is\n\t\t * selected at random from among the top RAND_DIM dimensions with the\n\t\t * highest variance. A value of 5 works well.\n\t\t */\n\t\tRAND_DIM=5\n\t};\n\n\tint TreeNum;\n\tint TreeNumBuild;\n\tint ml; //merge_level\n\tint max_deepth;\n\tint veclen_;\n\t//DataType* var_;\n\tomp_lock_t rootlock;\n\tbool error_flag;\n\t//DataType* mean_;\n\tstd::vector tree_roots_;\n\tstd::vector< std::pair > mlNodeList;\n\tstd::vector> LeafLists;\n\tUSING_BASECLASS_SYMBOLS\n\n\t//kgraph code\n\n\tstatic void GenRandom (std::mt19937& rng, unsigned *addr, unsigned size, unsigned N) {\n\t\tfor (unsigned i = 0; i < size; ++i) {\n\t\t\taddr[i] = rng() % (N - size);\n\t\t}\n\t\tstd::sort(addr, addr + size);\n\t\tfor (unsigned i = 1; i < size; ++i) {\n\t\t\tif (addr[i] <= addr[i-1]) {\n\t\t\t\taddr[i] = addr[i-1] + 1;\n\t\t\t}\n\t\t}\n\t\tunsigned off = rng() % N;\n\t\tfor (unsigned i = 0; i < size; ++i) {\n\t\t\taddr[i] = (addr[i] + off) % N;\n\t\t}\n\t}\n\n\n\tvoid DFSbuild(Node* node, std::mt19937& rng, unsigned* indices, unsigned count, unsigned offset){\n\t\t//omp_set_lock(&rootlock);\n\t\t//std::cout<treeid<<\":\"<DivDim = -1;\n\t\t\tnode->Lchild = NULL;\n\t\t\tnode->Rchild = NULL;\n\t\t\tnode->StartIdx = offset;\n\t\t\tnode->EndIdx = offset + count;\n\t\t\t//add points\n\n\t\t}else{\n\t\t\tunsigned idx;\n\t\t\tunsigned cutdim;\n\t\t\tDataType cutval;\n\t\t\tmeanSplit(rng, indices, count, idx, cutdim, cutval);\n\t\t\tnode->DivDim = cutdim;\n\t\t\tnode->DivVal = cutval;\n\t\t\tnode->StartIdx = offset;\n\t\t\tnode->EndIdx = offset + count;\n\t\t\tNode* nodeL = new Node(); Node* nodeR = new Node();\n\t\t\tnode->Lchild = nodeL;\n\t\t\tnodeL->treeid = node->treeid;\n\t\t\tDFSbuild(nodeL, rng, indices, idx, offset);\n\t\t\tnode->Rchild = nodeR;\n\t\t\tnodeR->treeid = node->treeid;\n\t\t\tDFSbuild(nodeR, rng, indices+idx, count-idx, offset+idx);\n\t\t}\n\t}\n\n\tvoid DFStest(unsigned level, unsigned dim, Node* node){\n\t\tif(node->Lchild !=NULL){\n\t\t\tDFStest(++level, node->DivDim, node->Lchild);\n\t\t\t//if(level > 15)\n\t\t\tstd::cout<<\"dim: \"<DivDim<<\"--cutval: \"<DivVal<<\"--S: \"<StartIdx<<\"--E: \"<EndIdx<<\" TREE: \"<treeid<Lchild->Lchild ==NULL){\n\t\t\t\tstd::vector& tmp = LeafLists[node->treeid];\n\t\t\t\tfor(unsigned i = node->Rchild->StartIdx; i < node->Rchild->EndIdx; i++)\n\t\t\t\t\tstd::cout<DivDim]<<\" \";\n\t\t\t\tstd::cout<Rchild !=NULL){\n\t\t\tDFStest(++level, node->DivDim, node->Rchild);\n\t\t}\n\t\telse{\n\t\t\tstd::cout<<\"dim: \"<& tmp = LeafLists[node->treeid];\n\t\t\tfor(unsigned i = node->StartIdx; i < node->EndIdx; i++)\n\t\t\t\tstd::cout< visited(N, false);\n\t\tknn_graph.resize(N);\n\t\tfor (auto &nhood: nhoods) {\n\t\t\t//nhood.nn_new.resize(params_.S * 2);\n\t\t\tnhood.pool.resize(params_.L+1);\n\t\t\tnhood.radius = std::numeric_limits::max();\n\t\t}\n\n\n\t\t//build tree\n\t\tstd::vector indices(N);\n\t\tLeafLists.resize(TreeNum);\n\t\tstd::vector ActiveSet;\n\t\tstd::vector NewSet;\n\t\tfor(unsigned i = 0; i < (unsigned)TreeNum; i++){\n\t\t\tNode* node = new Node;\n\t\t\tnode->DivDim = -1;\n\t\t\tnode->Lchild = NULL;\n\t\t\tnode->Rchild = NULL;\n\t\t\tnode->StartIdx = 0;\n\t\t\tnode->EndIdx = N;\n\t\t\tnode->treeid = i;\n\t\t\ttree_roots_.push_back(node);\n\t\t\tActiveSet.push_back(node);\n\t\t}\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < N; i++)indices[i] = i;\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < (unsigned)TreeNum; i++){\n\t\t\tstd::vector& myids = LeafLists[i];\n\t\t\tmyids.resize(N);\n\t\t\tstd::copy(indices.begin(), indices.end(),myids.begin());\n\t\t\tstd::random_shuffle(myids.begin(), myids.end());\n\t\t}\n\t\tomp_init_lock(&rootlock);\n\t\twhile(!ActiveSet.empty() && ActiveSet.size() < 1100){\n#pragma omp parallel for\n\t\t\tfor(unsigned i = 0; i < ActiveSet.size(); i++){\n\t\t\t\tNode* node = ActiveSet[i];\n\t\t\t\tunsigned mid;\n\t\t\t\tunsigned cutdim;\n\t\t\t\tDataType cutval;\n\t\t\t\tstd::mt19937 rng(seed ^ omp_get_thread_num());\n\t\t\t\tstd::vector& myids = LeafLists[node->treeid];\n\n\t\t\t\tmeanSplit(rng, &myids[0]+node->StartIdx, node->EndIdx - node->StartIdx, mid, cutdim, cutval);\n\n\t\t\t\tnode->DivDim = cutdim;\n\t\t\t\tnode->DivVal = cutval;\n\t\t\t\t//node->StartIdx = offset;\n\t\t\t\t//node->EndIdx = offset + count;\n\t\t\t\tNode* nodeL = new Node(); Node* nodeR = new Node();\n\t\t\t\tnodeR->treeid = nodeL->treeid = node->treeid;\n\t\t\t\tnodeL->StartIdx = node->StartIdx;\n\t\t\t\tnodeL->EndIdx = node->StartIdx+mid;\n\t\t\t\tnodeR->StartIdx = nodeL->EndIdx;\n\t\t\t\tnodeR->EndIdx = node->EndIdx;\n\t\t\t\tnode->Lchild = nodeL;\n\t\t\t\tnode->Rchild = nodeR;\n\t\t\t\tomp_set_lock(&rootlock);\n\t\t\t\tif(mid>params_.S)NewSet.push_back(nodeL);\n\t\t\t\tif(nodeR->EndIdx - nodeR->StartIdx > params_.S)NewSet.push_back(nodeR);\n\t\t\t\tomp_unset_lock(&rootlock);\n\t\t\t}\n\t\t\tActiveSet.resize(NewSet.size());\n\t\t\tstd::copy(NewSet.begin(), NewSet.end(),ActiveSet.begin());\n\t\t\tNewSet.clear();\n\t\t}\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < ActiveSet.size(); i++){\n\t\t\tNode* node = ActiveSet[i];\n\t\t\t//omp_set_lock(&rootlock);\n\t\t\t//std::cout<EndIdx-node->StartIdx<& myids = LeafLists[node->treeid];\n\t\t\tDFSbuild(node, rng, &myids[0]+node->StartIdx, node->EndIdx-node->StartIdx, node->StartIdx);\n\t\t}\n\t}\n void outputVisitBucketNum(){}\n\n\tvoid initGraph(){\n\t\t//initial\n\t\tunsigned N = features_.get_rows();\n\t\tunsigned seed = 1998;\n\t\tstd::mt19937 rng(seed);\n\t\tnhoods.resize(N);\n\t\tg.resize(N);\n\t\tboost::dynamic_bitset<> visited(N, false);\n\t\tknn_graph.resize(N);\n\t\tfor (auto &nhood: nhoods) {\n\t\t\t//nhood.nn_new.resize(params_.S * 2);\n\t\t\tnhood.pool.resize(params_.L+1);\n\t\t\tnhood.radius = std::numeric_limits::max();\n\t\t}\n\n\n\t\t//build tree\n\t\tstd::vector indices(N);\n\t\tLeafLists.resize(TreeNum);\n\t\tstd::vector ActiveSet;\n\t\tstd::vector NewSet;\n\t\tfor(unsigned i = 0; i < (unsigned)TreeNum; i++){\n\t\t\tNode* node = new Node;\n\t\t\tnode->DivDim = -1;\n\t\t\tnode->Lchild = NULL;\n\t\t\tnode->Rchild = NULL;\n\t\t\tnode->StartIdx = 0;\n\t\t\tnode->EndIdx = N;\n\t\t\tnode->treeid = i;\n\t\t\ttree_roots_.push_back(node);\n\t\t\tActiveSet.push_back(node);\n\t\t}\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < N; i++)indices[i] = i;\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < (unsigned)TreeNum; i++){\n\t\t\tstd::vector& myids = LeafLists[i];\n\t\t\tmyids.resize(N);\n\t\t\tstd::copy(indices.begin(), indices.end(),myids.begin());\n\t\t\tstd::random_shuffle(myids.begin(), myids.end());\n\t\t}\n\t\tomp_init_lock(&rootlock);\n\t\twhile(!ActiveSet.empty() && ActiveSet.size() < 1100){\n#pragma omp parallel for\n\t\t\tfor(unsigned i = 0; i < ActiveSet.size(); i++){\n\t\t\t\tNode* node = ActiveSet[i];\n\t\t\t\tunsigned mid;\n\t\t\t\tunsigned cutdim;\n\t\t\t\tDataType cutval;\n\t\t\t\tstd::mt19937 rng(seed ^ omp_get_thread_num());\n\t\t\t\tstd::vector& myids = LeafLists[node->treeid];\n\n\t\t\t\tmeanSplit(rng, &myids[0]+node->StartIdx, node->EndIdx - node->StartIdx, mid, cutdim, cutval);\n\n\t\t\t\tnode->DivDim = cutdim;\n\t\t\t\tnode->DivVal = cutval;\n\t\t\t\t//node->StartIdx = offset;\n\t\t\t\t//node->EndIdx = offset + count;\n\t\t\t\tNode* nodeL = new Node(); Node* nodeR = new Node();\n\t\t\t\tnodeR->treeid = nodeL->treeid = node->treeid;\n\t\t\t\tnodeL->StartIdx = node->StartIdx;\n\t\t\t\tnodeL->EndIdx = node->StartIdx+mid;\n\t\t\t\tnodeR->StartIdx = nodeL->EndIdx;\n\t\t\t\tnodeR->EndIdx = node->EndIdx;\n\t\t\t\tnode->Lchild = nodeL;\n\t\t\t\tnode->Rchild = nodeR;\n\t\t\t\tomp_set_lock(&rootlock);\n\t\t\t\tif(mid>params_.S)NewSet.push_back(nodeL);\n\t\t\t\tif(nodeR->EndIdx - nodeR->StartIdx > params_.S)NewSet.push_back(nodeR);\n\t\t\t\tomp_unset_lock(&rootlock);\n\t\t\t}\n\t\t\tActiveSet.resize(NewSet.size());\n\t\t\tstd::copy(NewSet.begin(), NewSet.end(),ActiveSet.begin());\n\t\t\tNewSet.clear();\n\t\t}\n#pragma omp parallel for\n\t\tfor(unsigned i = 0; i < ActiveSet.size(); i++){\n\t\t\tNode* node = ActiveSet[i];\n\t\t\t//omp_set_lock(&rootlock);\n\t\t\t//std::cout<EndIdx-node->StartIdx<& myids = LeafLists[node->treeid];\n\t\t\tDFSbuild(node, rng, &myids[0]+node->StartIdx, node->EndIdx-node->StartIdx, node->StartIdx);\n\t\t}\n\t\t//DFStest(0,0,tree_roots_[0]);\n\t\t//build tree completed\n\n\t\tfor(size_t i = 0; i < (unsigned)TreeNumBuild; i++){\n\t\t\tgetMergeLevelNodeList(tree_roots_[i], i ,0);\n\t\t}\n\n#pragma omp parallel for\t\n\t\tfor(size_t i = 0; i < mlNodeList.size(); i++){\n\t\t\tmergeSubGraphs(mlNodeList[i].second, mlNodeList[i].first);\n\t\t}\n\n\n#pragma omp parallel\n\t\t{\n#ifdef _OPENMP\n\t\t\tstd::mt19937 rng(seed ^ omp_get_thread_num());\n#else\n\t\t\tstd::mt19937 rng(seed);\n#endif\n\t\t\tstd::vector random(params_.S + 1);\n\n#pragma omp for\n\t\t\tfor (unsigned n = 0; n < N; ++n) {\n\t\t\t\tauto &nhood = nhoods[n];\n\t\t\t\tPoints &pool = nhood.pool;\n\t\t\t\tif(nhood.nn_new.size()compare(\n\t\t\t\t\t\t\tfeatures_.get_row(n), features_.get_row(rand_id), features_.get_cols());\n\t\t\t\t\tCandidate c(rand_id,dist);\n\t\t\t\t\tknn_graph[n].insert(c);\n\t\t\t\t}\n\n\t\t\t\t//omp_set_lock(&rootlock);\n\t\t\t\t//if(knn_graph[n].size() < nhood.L)std::cout<row_id;//random[i++];\n\t\t\t\t\tnhood.nn_new[l] = it->row_id;\n\t\t\t\t\tnn.dist = it->distance;//distance_->compare(features_.get_row(n), features_.get_row(nn.id), features_.get_cols());\n\t\t\t\t\tnn.flag = true;it++;\n\t\t\t\t\t//if(it == knn_graph[n].rend())break;\n\t\t\t\t}\n\t\t\t\tsort(pool.begin(), pool.begin() + nhood.L);\n\t\t\t}\n\t\t}\n\t\tknn_graph.clear();\n#ifdef INFO\n\t\tstd::cout<<\"initial completed\"<. All Rights Reserved.\n#ifndef EFANNA\n#define EFANNA\n#include \"general/distance.hpp\"\n#include \"general/matrix.hpp\"\n#include \"general/params.hpp\"\n#include \"algorithm/init_indices.hpp\"\nnamespace efanna{\ntemplate \nclass FIndex{\npublic:\n\ttypedef InitIndex IndexType;\n\n\tFIndex(const Matrix& features, Distance* d, const IndexParams& params)\n\t: index_params_(params)\n\t{\n\t\tinit_algorithm init_index_type= params.init_index_type;\n\t\tindex_params_ = params;\n\t\tinitIndex_ = create_index_by_type(init_index_type, features, params, d);\n\t}\n\n\tvirtual ~FIndex () {\n\t}\n\tvoid buildIndex()\n\t{\n\t\tinitIndex_->buildIndex();\n\t}\n\tvoid buildTrees()\n\t{\n\t\tinitIndex_->buildTrees();\n\t}\n\tvoid knnSearch(int k, const Matrix& query){\n\t\tinitIndex_->knnSearch(k, query);\n\t}\n\tvoid saveIndex(char* filename){\n\t\tinitIndex_->saveIndex(filename);\n\t}\n\tvoid loadIndex(char* filename){\n\t\tinitIndex_->loadIndex(filename);\n\t}\n\tvoid saveTrees(char* filename){\n\t\tinitIndex_->saveTrees(filename);\n\t}\n\tvoid loadTrees(char* filename){\n\t\tinitIndex_->loadTrees(filename);\n\t}\n\tvoid saveGraph(char* filename){\n\t\tinitIndex_->saveGraph(filename);\n\t}\n\tvoid loadGraph(char* filename){\n\t\tinitIndex_->loadGraph(filename);\n\t}\n\tvoid saveResults(char* filename){\n\t\tinitIndex_->saveResults(filename);\n\t}\n\tvoid setSearchParams(int epochs, int init_num, int extend_to, int search_trees = 0, int search_lv=-1, int search_method = 0){\n\t\tinitIndex_->setSearchParams(epochs, init_num, extend_to,search_trees, search_lv, search_method);\n\t}\n\tsize_t getGraphSize(){\n\t\treturn initIndex_->getGraphSize();\n\t}\n\tstd::vector getGraphRow(unsigned row_id){\n\t\treturn initIndex_->getGraphRow(row_id);\n\t}\n\tvoid outputVisitBucketNum(){\n\t\tinitIndex_->outputVisitBucketNum();\n\t}\n\nprivate:\n\t/** Pointer to actual index class */\n\tIndexType* initIndex_;\n\t/** Parameters passed to the index */\n\tIndexParams index_params_;\n\n};\n}\n#endif\n" }, { "path": "general/distance.hpp", "content": "#ifndef EFANNA_DISTANCE_H_\n#define EFANNA_DISTANCE_H_\n#include \n#include \n#include \n#include \n#include \n#include //for debug\n\n#ifdef __GNUC__\n#ifdef __AVX__\n#define KGRAPH_MATRIX_ALIGN 32\n#else\n#ifdef __SSE2__\n#define KGRAPH_MATRIX_ALIGN 16\n#else\n#define KGRAPH_MATRIX_ALIGN 4\n#endif\n#endif\n#endif\n\n\n\nnamespace efanna {\n\ntemplate\nstruct Candidate {\n size_t row_id;\n T distance;\n Candidate(const size_t row_id, const T distance): row_id(row_id), distance(distance) { }\n\n bool operator >(const Candidate& rhs) const {\n if (this->distance == rhs.distance) {\n return this->row_id > rhs.row_id;\n }\n return this->distance > rhs.distance;\n }\n bool operator <(const Candidate& rhs) const {\n if (this->distance == rhs.distance) {\n return this->row_id < rhs.row_id;\n }\n return this->distance < rhs.distance;\n }\n};\n\ntemplate\nclass Distance {\npublic:\n virtual T compare(const T* a, const T* b, size_t length) const = 0;\n virtual T norm(const T* a, size_t length) const = 0;\n virtual T dot(const T* a, const T* b, size_t length) const = 0;\n virtual ~Distance() {}\n};\n\n#define SSE_L2SQR(addr1, addr2, dest, tmp1, tmp2) \\\n tmp1 = _mm_load_ps(addr1);\\\n tmp2 = _mm_load_ps(addr2);\\\n tmp1 = _mm_sub_ps(tmp1, tmp2); \\\n tmp1 = _mm_mul_ps(tmp1, tmp1); \\\n dest = _mm_add_ps(dest, tmp1);\n\n\ntemplate\nclass L2DistanceSSE: public Distance {\npublic:\n typedef T ResultType;\n /**\n * \n * We use msse intrinstic here, we should ensure data align.\n */\nResultType compare(const T* a, const T* b, size_t size) const {\n\n __m128 sum;\n __m128 l0, l1, l2, l3;\n __m128 r0, r1, r2, r3;\n unsigned D = (size + 3) & ~3U;\n unsigned DR = D % 16;\n unsigned DD = D - DR;\n const float *l = a;\n const float *r = b;\n const float *e_l = l + DD;\n const float *e_r = r + DD;\n float unpack[4] __attribute__ ((aligned (16))) = {0, 0, 0, 0};\n ResultType ret = 0.0;\n sum = _mm_load_ps(unpack);\n switch (DR) {\n case 12:\n SSE_L2SQR(e_l+8, e_r+8, sum, l2, r2);\n case 8:\n SSE_L2SQR(e_l+4, e_r+4, sum, l1, r1);\n case 4:\n SSE_L2SQR(e_l, e_r, sum, l0, r0);\n }\n for (unsigned i = 0; i < DD; i += 16, l += 16, r += 16) {\n SSE_L2SQR(l, r, sum, l0, r0);\n SSE_L2SQR(l + 4, r + 4, sum, l1, r1);\n SSE_L2SQR(l + 8, r + 8, sum, l2, r2);\n SSE_L2SQR(l + 12, r + 12, sum, l3, r3);\n }\n _mm_storeu_ps(unpack, sum);\n ret = unpack[0] + unpack[1] + unpack[2] + unpack[3];\n return ret;//sqrt(ret);\n}\n\nT norm(const T* a, size_t length) const\n{\n\treturn 0;\n}\nT dot(const T* a, const T* b, size_t length) const\n{\n\treturn 0;\n}\n\n \n};\n\ntemplate\nclass L2Distance: public Distance {\npublic:\n typedef T ResultType;\n /**\n * Copied from flann\n * We do not want msse intrinstic here to avoid misalign problems.\n */\n ResultType compare(const T* a, const T* b, size_t size) const {\n ResultType result = ResultType();\n ResultType diff0, diff1, diff2, diff3;\n const T* last = a + size;\n const T* lastgroup = last - 3;\n\n /* Process 4 items with each loop for efficiency. */\n while (a < lastgroup) {\n diff0 = (ResultType)(a[0] - b[0]);\n diff1 = (ResultType)(a[1] - b[1]);\n diff2 = (ResultType)(a[2] - b[2]);\n diff3 = (ResultType)(a[3] - b[3]);\n result += diff0 * diff0 + diff1 * diff1 + diff2 * diff2 + diff3 * diff3;\n a += 4;\n b += 4;\n }\n /* Process last 0-3 pixels. Not needed for standard vector lengths. */\n while (a < last) {\n diff0 = (ResultType)(*a++ - *b++);\n result += diff0 * diff0;\n }\n return result;//sqrt(result);\n }\nT norm(const T* a, size_t length) const\n{\n return 0;\n}\nT dot(const T* a, const T* b, size_t length) const\n{\n return 0;\n}\n};\n\ntemplate\nclass L2DistanceAVXr4: public Distance {\npublic:\n typedef T ResultType;\n /**\n * \n * We use msse intrinstic here, we should ensure data align.\n */\n#define AVX_L2SQR(addr1, addr2, dest, tmp1, tmp2) \\\n tmp1 = _mm256_loadu_ps(addr1);\\\n tmp2 = _mm256_loadu_ps(addr2);\\\n tmp1 = _mm256_sub_ps(tmp1, tmp2); \\\n tmp1 = _mm256_mul_ps(tmp1, tmp1); \\\n dest = _mm256_add_ps(dest, tmp1);\nResultType compare(const T* a, const T* b, size_t size) const{\n/*\n __m256 sum;\n __m256 l0, l1;\n __m256 r0, r1;\n unsigned D = (size + 7) & ~7U;\n unsigned DR = D % 16;\n unsigned DD = D - DR;\n const float *l = a;\n const float *r = b;\n const float *e_l = l + DD;\n const float *e_r = r + DD;\n float unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n ResultType ret = 0.0;\n sum = _mm256_loadu_ps(unpack);\n if(DR){AVX_L2SQR(e_l, e_r, sum, l0, r0);}\n \n for (unsigned i = 0; i < DD; i += 16, l += 16, r += 16) {\n\tAVX_L2SQR(l, r, sum, l0, r0);\n\tAVX_L2SQR(l + 8, r + 8, sum, l1, r1);\n }\n _mm256_storeu_ps(unpack, sum);\n ret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n return ret;//sqrt(ret);\n*/\n __m256 sum;\n __m256 l0, l1, l2, l3;\n __m256 r0, r1, r2, r3;\n unsigned D = (size + 7) & ~7U;\n unsigned DR = D % 32;\n unsigned DD = D - DR;\n const float *l = a;\n const float *r = b;\n const float *e_l = l + DD;\n const float *e_r = r + DD;\n float unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n ResultType ret = 0.0;\n sum = _mm256_loadu_ps(unpack);\n switch (DR) {\n case 24:\n AVX_L2SQR(e_l+16, e_r+16, sum, l2, r2);\n case 16:\n AVX_L2SQR(e_l+8, e_r+8, sum, l1, r1);\n case 8:\n AVX_L2SQR(e_l, e_r, sum, l0, r0);\n }\n for (unsigned i = 0; i < DD; i += 32, l += 32, r += 32) {\n AVX_L2SQR(l, r, sum, l0, r0);\n AVX_L2SQR(l + 8, r + 8, sum, l1, r1);\n AVX_L2SQR(l + 16, r + 16, sum, l2, r2);\n AVX_L2SQR(l + 24, r + 24, sum, l3, r3);\n }\n _mm256_storeu_ps(unpack, sum);\n ret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n return ret;\n}\n#define AVX_L2NORM(addr, dest, tmp) \\\n tmp = _mm256_loadu_ps(addr); \\\n tmp = _mm256_mul_ps(tmp, tmp); \\\n dest = _mm256_add_ps(dest, tmp);\n T norm(const T* a, size_t size) const\n {\n\t__m256 sum;\n \t__m256 l0, l1;\n \tunsigned D = (size + 7) & ~7U;\n \tunsigned DR = D % 16;\n \tunsigned DD = D - DR;\n \tconst float *l = a;\n \tconst float *e_l = l + DD;\n \tfloat unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n \tResultType ret = 0.0;\n \tsum = _mm256_loadu_ps(unpack);\n \tif(DR){AVX_L2NORM(e_l, sum, l0);}\n\tfor (unsigned i = 0; i < DD; i += 16, l += 16) {\n\t AVX_L2NORM(l, sum, l0);\n\t AVX_L2NORM(l + 8, sum, l1);\n }\n \t_mm256_storeu_ps(unpack, sum);\n \tret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n \treturn ret;\n }\n#define AVX_L2DOT(addr1, addr2, dest, tmp1, tmp2) \\\n tmp1 = _mm256_loadu_ps(addr1);\\\n tmp2 = _mm256_loadu_ps(addr2);\\\n tmp1 = _mm256_mul_ps(tmp1, tmp2); \\\n dest = _mm256_add_ps(dest, tmp1);\n T dot(const T* a, const T* b, size_t size) const\n {\n\t__m256 sum;\n \t __m256 l0, l1;\n \t __m256 r0, r1;\n \tunsigned D = (size + 7) & ~7U;\n \tunsigned DR = D % 16;\n \tunsigned DD = D - DR;\n \tconst float *l = a;\n \tconst float *r = b;\n \tconst float *e_l = l + DD;\n \tconst float *e_r = r + DD;\n \tfloat unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n \tResultType ret = 0.0;\n \tsum = _mm256_loadu_ps(unpack);\n \tif(DR){AVX_L2DOT(e_l, e_r, sum, l0, r0);}\n \n \tfor (unsigned i = 0; i < DD; i += 16, l += 16, r += 16) {\n\t AVX_L2DOT(l, r, sum, l0, r0);\n\t AVX_L2DOT(l + 8, r + 8, sum, l1, r1);\n \t}\n \t_mm256_storeu_ps(unpack, sum);\n \tret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n \treturn ret;\n }\n};\n\ntemplate\nclass L2DistanceAVX: public Distance {\npublic:\n typedef T ResultType;\n /**\n * \n * We use msse intrinstic here, we should ensure data align.\n */\n#define AVX_L2SQR(addr1, addr2, dest, tmp1, tmp2) \\\n tmp1 = _mm256_loadu_ps(addr1);\\\n tmp2 = _mm256_loadu_ps(addr2);\\\n tmp1 = _mm256_sub_ps(tmp1, tmp2); \\\n tmp1 = _mm256_mul_ps(tmp1, tmp1); \\\n dest = _mm256_add_ps(dest, tmp1);\nResultType compare(const T* a, const T* b, size_t size) const{\n\n __m256 sum;\n __m256 l0, l1;\n __m256 r0, r1;\n unsigned D = (size + 7) & ~7U;\n unsigned DR = D % 32;\n unsigned DD = D - DR;\n const float *l = a;\n const float *r = b;\n const float *e_l = l + DD;\n const float *e_r = r + DD;\n float unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n ResultType ret = 0.0;\n sum = _mm256_loadu_ps(unpack);\n if(DR){AVX_L2SQR(e_l, e_r, sum, l0, r0);}\n \n for (unsigned i = 0; i < DD; i += 16, l += 16, r += 16) {\n\tAVX_L2SQR(l, r, sum, l0, r0);\n\tAVX_L2SQR(l + 8, r + 8, sum, l1, r1);\n }\n _mm256_storeu_ps(unpack, sum);\n ret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n return ret;//sqrt(ret);\n}\n#define AVX_L2NORM(addr, dest, tmp) \\\n tmp = _mm256_loadu_ps(addr); \\\n tmp = _mm256_mul_ps(tmp, tmp); \\\n dest = _mm256_add_ps(dest, tmp);\n T norm(const T* a, size_t size) const\n {\n\t__m256 sum;\n \t__m256 l0, l1;\n \tunsigned D = (size + 7) & ~7U;\n \tunsigned DR = D % 16;\n \tunsigned DD = D - DR;\n \tconst float *l = a;\n \tconst float *e_l = l + DD;\n \tfloat unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n \tResultType ret = 0.0;\n \tsum = _mm256_loadu_ps(unpack);\n \tif(DR){AVX_L2NORM(e_l, sum, l0);}\n\tfor (unsigned i = 0; i < DD; i += 16, l += 16) {\n\t AVX_L2NORM(l, sum, l0);\n\t AVX_L2NORM(l + 8, sum, l1);\n }\n \t_mm256_storeu_ps(unpack, sum);\n \tret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n \treturn ret;\n }\n#define AVX_L2DOT(addr1, addr2, dest, tmp1, tmp2) \\\n tmp1 = _mm256_loadu_ps(addr1);\\\n tmp2 = _mm256_loadu_ps(addr2);\\\n tmp1 = _mm256_mul_ps(tmp1, tmp2); \\\n dest = _mm256_add_ps(dest, tmp1);\n T dot(const T* a, const T* b, size_t size) const\n {\n\t__m256 sum;\n \t __m256 l0, l1;\n \t __m256 r0, r1;\n \tunsigned D = (size + 7) & ~7U;\n \tunsigned DR = D % 16;\n \tunsigned DD = D - DR;\n \tconst float *l = a;\n \tconst float *r = b;\n \tconst float *e_l = l + DD;\n \tconst float *e_r = r + DD;\n \tfloat unpack[8] __attribute__ ((aligned (32))) = {0, 0, 0, 0, 0, 0, 0, 0};\n \tResultType ret = 0.0;\n \tsum = _mm256_loadu_ps(unpack);\n \tif(DR){AVX_L2DOT(e_l, e_r, sum, l0, r0);}\n \n \tfor (unsigned i = 0; i < DD; i += 16, l += 16, r += 16) {\n\t AVX_L2DOT(l, r, sum, l0, r0);\n\t AVX_L2DOT(l + 8, r + 8, sum, l1, r1);\n \t}\n \t_mm256_storeu_ps(unpack, sum);\n \tret = unpack[0] + unpack[1] + unpack[2] + unpack[3] + unpack[4] + unpack[5] + unpack[6] + unpack[7];\n \treturn ret;\n }\n};\n}\n#endif\n" }, { "path": "general/matrix.hpp", "content": "#ifndef EFANNA_MATRIX_H\n#define EFANNA_MATRIX_H\n\n#include \n#include \n#include \n#include \n#include \n#include \"distance.hpp\"\nnamespace efanna {\n\ntemplate \nclass Matrix {\npublic:\n /**\n * Create a matrix using number of rows and cols with rawdata.\n * rawdata should be compact without any alignments.\n * Notice that efnn::Matrix itself just keep a reference of data and\n * does not make a copy of data, so keep it available.\n */\n Matrix(const size_t rows, const size_t cols, const void* data):\n rows_(rows), cols_(cols) {\n size_t align_cols;\n#ifdef __GNUC__\n#ifdef __AVX__\n align_cols = (cols + 7)/8*8;//re align to sse format\n#else\n#ifdef __SSE2__\n align_cols = (cols + 3)/4*4;\n#else\n align_cols = cols;\n#endif\n#endif\n#endif\n //std::cout<<\" DD: \"<(data) + (align_cols * i));\n }\n }\n\n size_t get_cols() const {\n return cols_;\n }\n\n size_t get_rows() const {\n return rows_;\n }\n\n const T* get_row(const size_t index) const {\n if (index >= rows_) {\n throw std::runtime_error(\"index out of range\");\n }\n return row_pointers_[index];\n }\n\n // Debug usage only\n std::vector > brute_force_search(size_t idx, size_t k, Distance* distance) const {\n printf(\"idx: %lu\\n\", idx);\n std::vector > result;\n for (size_t i = 0; i < rows_; i++) {\n result.push_back(std::make_pair(\n distance->compare(get_row(i), get_row(idx), cols_),\n i));\n }\n std::partial_sort(result.begin(), result.begin() + k, result.end());\n result.resize(k);\n return result;\n }\nprivate:\n size_t rows_, cols_;\n std::vector row_pointers_;\n};\n\n}\n#endif\n" }, { "path": "general/params.hpp", "content": "#ifndef EFANNA_PARAMS_H_\n#define EFANNA_PARAMS_H_\n#include \n#include \n\nnamespace efanna {\n enum init_algorithm {\n KDTREE_UB,\n\t HASHING\n };\n\n union ValueType{\n int int_val;\n float float_val;\n char* str_pt;\n };\n\n typedef std::map ExtraParamsMap;\n struct IndexParams{\n\n init_algorithm init_index_type;\n size_t K; //build knn table with nn = K\n size_t S; //nn sets' max size\n size_t L =30;//rnn size\n size_t TNS = 10;//tree node size\n size_t Check_K = 40;\n int build_epoches;\n int extend_num; //number to extend each time\n size_t pool_size = 100;\n size_t init_num = 100;\n ExtraParamsMap extra_params;\n bool reverse_nn_used;\n };\n\n struct SearchParams{\n int search_init_num;\n int search_epoches;\n int search_method;\n unsigned extend_to;\n unsigned tree_num;\n unsigned search_depth;\n };\n\n}\n#endif\n" }, { "path": "matlab/.gitignore", "content": "*.mexa32\n*.mexa64\n*.mexw32\n*.mexw64\n*.mexmac\n" }, { "path": "matlab/README.md", "content": "**Compilation**\n\n1. Compile with: ``mex CXXFLAGS=\"\\$CXXFLAGS -std=c++11 -O3 -march=native -fopenmp -Wall -lboost_timer -lboost_system\" LDFLAGS=\"\\$LDFLAGS -fopenmp\" findex.cc -I../ -largeArrayDims``\n\n2. Run any matlab program with efanna! \n\n-----\n\n**Dependencies**\n\n* Matlab 2010b or above.\n* Other prerequisites are the same as C++ prerequisites, see \n\n-----\n\n**Examples**\n\n* Example programs are provided under folder \"samples\". \n\n* For instance, use ``matlab -nodesktop -nosplash -r \"run('./samples/example_buildgraph')\"`` to try. (Don't forget to provide inputs! Default inputs are placed under ``~/data/sift/``. You may change this path in the \\*.m programs)\n\n* Every sample does the same job as C++ programs under ``../samples/`` do, and their parameters are mostly identical.\n\n* Additionaly, a *row-wise* organized sparse matrix describing nearest neighbours for all points is returned from functions for building or loading graphs\n\n* Look inside samples for more API details. \n\n-----\n\n**FAQ**\n\nCommon errors and their solutions:\n\n* If error ``redeclaration of C++ built-in type ‘char16_t’`` occurs in compilation\n * Try uncommenting lines starts with ``define`` and ``undef`` at the beginning of BOTH ``findex.cc`` and ``handle_wrapper.hpp``.\n * Compile again. Problem solved.\n * It may indicates you are using very old version of matlab and not fitting your g++ version well.\n\n* If error related to ``{mablab root}/sys/os/glnxa64/libstdc++.so.6`` occurs in runtime (typically, ``version 'CXXABI_1.x.x' not found (required by ...)``, or `` version `GLIBCXX_3.x.x' not found (required by ...)``\n * Try ``export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libstdc++.so.6``. You may need to rewrite the command with your own path of ``libstdc++.so.6`` according to the place you setup your gcc compiler's lib.\n * OR, directly substitute ``/usr/local/MATLAB/{your version}/sys/os/glnx64/libstdc++.so.6`` with your compiler's ``libstdc++.so.6``. You may need to rewrite the command with the place you setup your matlab.\n * It indicates your mex compiler automatically links the g++ lib under your matlab directory, which is not compatible with your g++ compiler, during the compilation step. \n" }, { "path": "matlab/efanna.m", "content": "classdef efanna < handle\n properties (SetAccess = private, Hidden = true)\n objectHandle;\n index_name;\n end\n methods\n % Constructor\n function this = efanna(data, varargin)\n this.objectHandle = findex('new', data, 'kdtreeub', 'l2', varargin{:});\n this.index_name = 'kdtreeub';\n end\n % Destructor\n function delete(this)\n findex('destruct', this.objectHandle);\n end\n \n function graph_mat = build_index(this)\n graph_mat = findex('build_index', this.objectHandle);\n end\n\n function build_trees(this)\n findex('build_trees', this.objectHandle);\n end\n\n function load_index(this, filename)\n findex('load_index', this.objectHandle, filename);\n end\n\n function save_index(this, filename)\n findex('save_index', this.objectHandle, filename);\n end\n\n function graph_mat = load_graph(this, filename)\n graph_mat = findex('load_graph', this.objectHandle, filename);\n end\n\n function save_graph(this, filename)\n findex('save_graph', this.objectHandle, filename);\n end\n\n function load_trees(this, filename)\n findex('load_trees', this.objectHandle, filename);\n end\n\n function save_trees(this, filename)\n findex('save_trees', this.objectHandle, filename);\n end\n % set search params (different index may share the same knn_search with different params)\n function set_search_params(this, varargin)\n findex('set_search_params', this.objectHandle, this.index_name, varargin{:});\n end\n\n function knn_search(this, k, query)\n findex('knn_search', this.objectHandle, k, query);\n end\n\n function save_result(this, filename)\n findex('save_result', this.objectHandle, filename);\n end\n end\nend\n" }, { "path": "matlab/findex.cc", "content": "#include \n#include \"handle_wrapper.hpp\"\n#include \n#include \n#include \n\n//#define char16_t LIBRARY_char16_t\n#include \n//#undef char16_t\n\nusing namespace efanna;\n\ntemplate\nMatrix copy_matrix(const mxArray *array)\n{\n int points_num = mxGetN(array);\n int dim = mxGetM(array);\n size_t mem_size = mxGetN(array)*mxGetM(array)*sizeof(T);\n void* data = malloc(mem_size);\n memcpy(data, mxGetData(array), mem_size);\n return Matrix(points_num, dim, (T*)data);\n}\n\ntemplate\nstruct construct_func {\n typedef FIndex* (*entry)(Matrix, Distance*, int, const mxArray**);\n};\n\n//TODO: default params\ntemplate\nFIndex* _construct_kdtreeub(Matrix dataset, Distance* dist, int in_n, const mxArray *in_array[]){\n if (in_n!=7 && in_n!=8 && in_n!=1) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n\n bool rnn_used = true;\n int trees = (int)(*mxGetPr(in_array[0]));\n if (in_n==1) {\n mexPrintf(\"kdtreeub params : %d\\n\", trees);\n return new FIndex(dataset, dist, KDTreeUbIndexParams(rnn_used, trees, 10, 10, 10, 10, 10, trees, 10));\n }\n\n int mlevel = (int)(*mxGetPr(in_array[1]));\n int epoches = (int)(*mxGetPr(in_array[2]));\n int L = (int)(*mxGetPr(in_array[3]));\n int check_k = (int)(*mxGetPr(in_array[4]));\n int K = (int)(*mxGetPr(in_array[5]));\n int S = (int)(*mxGetPr(in_array[6]));\n if (in_n==7) {\n mexPrintf(\"kdtreeub params : %d %d %d %d %d %d %d\\n\", trees, mlevel, epoches, L, check_k, K, S);\n return new FIndex(dataset, dist, KDTreeUbIndexParams(rnn_used, trees, mlevel, epoches, check_k, L, K, trees, S));\n } else if (in_n==8) {\n int build_trees = (int)(*mxGetPr(in_array[7]));\n mexPrintf(\"kdtreeub params : %d %d %d %d %d %d %d %d\\n\", trees, mlevel, epoches, L, check_k, K, S, build_trees);\n return new FIndex(dataset, dist, KDTreeUbIndexParams(rnn_used, trees, mlevel, epoches, check_k, L, K, build_trees, S));\n } \n return NULL; // ERROR if this line is reached\n}\n\n\ntemplate\nvoid _construct(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n<3) || (!mxIsChar(in_array[1])) || (!mxIsChar(in_array[2]))) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n if (out_n!=1) {\n mexErrMsgTxt(\"Incorrect number of output arguments\");\n }\n\n std::map* > dist_table = {\n {\"l2\", new L2DistanceAVX() }\n };\n std::map::entry> index_table = {\n {\"kdtreeub\", &_construct_kdtreeub },\n {\"nndescent\", &_construct_kdtreeub },\n {\"nnexp\", &_construct_kdtreeub }\n };\n\n std::string index_name = mxArrayToString(in_array[1]);\n if (index_table.find(index_name)==index_table.end()) {\n mexErrMsgTxt(\"Error: bad distance selector: wrong distance name.\");\n }\n std::string dist_name = mxArrayToString(in_array[2]);\n if (dist_table.find(dist_name)==dist_table.end()) {\n mexErrMsgTxt(\"Error: bad distance selector: wrong distance name.\");\n }\n\n //TODO: type check\n Matrix dataset = copy_matrix(in_array[0]);\n\n FIndex* result = (*index_table[index_name])(dataset, dist_table[dist_name], in_n-3, in_array+3);\n out_array[0] = handle2mat >(result);\n //mxFree(index_name);\n //mxFree(dist_name);\n}\n\ntemplate\nvoid _destruct(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if (in_n !=1) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n delete_wrapper >(in_array[0]);\n}\n\ntemplate\nvoid _get_graph_mat(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n //TODO: verify there're no bugs\n FIndex* handle = mat2handle >(in_array[0]);\n size_t nrows = handle->getGraphSize();\n std::map*> col_graph;\n int maxnnz = 0;\n for (unsigned i = 0; i < nrows; i++) {\n std::vector nodes = handle->getGraphRow(i);\n maxnnz += nodes.size();\n for (unsigned x : nodes) {\n if (col_graph.find(x) == col_graph.end()) {\n col_graph[x] = new std::vector();\n }\n col_graph[x]->push_back(i);\n }\n }\n \n //the type must be double*\n double* pr = (double *)mxCalloc(maxnnz, sizeof(double));\n mwIndex* ir = (size_t *)mxCalloc(maxnnz, sizeof(mwIndex));\n mwIndex* jc = (size_t *)mxCalloc(nrows + 1, sizeof(mwIndex));\n int nfilled = -1;\n int njc = 0;\n jc[0] = 0;\n for (unsigned i = 0; i < nrows; i++) {\n if (col_graph.find(i) == col_graph.end()) {\n njc ++;\n jc[njc] = jc[njc-1];\n continue;\n }\n for (unsigned x: *col_graph[i]) {\n nfilled ++;\n pr[nfilled] = 1;\n ir[nfilled] = x;\n }\n njc ++;\n jc[njc] = jc[njc-1] + col_graph[i]->size();\n delete col_graph[i];\n }\n\n out_array[0] = mxCreateSparse(nrows, nrows, maxnnz, mxREAL);\n mxSetPr(out_array[0], pr);\n mxSetIr(out_array[0], ir);\n mxSetJc(out_array[0], jc);\n}\n\ntemplate\nvoid _build_index(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if (in_n != 1) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n\n FIndex* handle = mat2handle >(in_array[0]);\n handle->buildIndex();\n _get_graph_mat(out_n, out_array, in_n, in_array);\n}\n\ntemplate\nvoid _build_trees(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if (in_n != 1) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n\n FIndex* handle = mat2handle >(in_array[0]);\n handle->buildTrees();\n}\n\ntemplate\nvoid _load_index(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->loadIndex(path);\n mxFree(path);\n}\n\ntemplate\nvoid _save_index(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->saveIndex(path);\n mxFree(path);\n}\n\ntemplate\nvoid _load_graph(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->loadGraph(path);\n _get_graph_mat(out_n, out_array, in_n, in_array);\n mxFree(path);\n}\n\ntemplate\nvoid _save_graph(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->saveGraph(path);\n mxFree(path);\n}\n\ntemplate\nvoid _load_trees(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->loadTrees(path);\n mxFree(path);\n}\n\ntemplate\nvoid _save_trees(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->saveTrees(path);\n mxFree(path);\n}\n\ntemplate\nvoid _set_search_params_kdtreeub(FIndex* handle, int in_n, const mxArray *in_array[]) {\n int search_trees = (int)(*mxGetPr(in_array[0]));\n int search_epoc = (int)(*mxGetPr(in_array[1]));\n int poolsz = (int)(*mxGetPr(in_array[2]));\n int search_extend = (int)(*mxGetPr(in_array[3]));\n int search_method = (int)(*mxGetPr(in_array[4]));\n handle->setSearchParams(search_epoc, poolsz, search_extend, search_trees, search_method);\n}\n\ntemplate\nvoid _set_search_params_nndescent(FIndex* handle, int in_n, const mxArray *in_array[]) {\n mexPrintf(\"No param needs to be set.\\n\");\n}\n\ntemplate\nvoid _set_search_params_nnexp(FIndex* handle, int in_n, const mxArray *in_array[]) {\n mexPrintf(\"No param needs to be set.\\n\");\n}\n\ntemplate\nvoid _set_search_params(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n < 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Incorrect number of input arguments\");\n }\n\n //XXX: why this does not need template type for the function?\n typedef void (*funcp)(FIndex*, int, const mxArray**);\n std::map index_table = {\n {\"kdtreeub\", &_set_search_params_kdtreeub},\n {\"nndescent\", &_set_search_params_nndescent},\n {\"nnexp\", &_set_search_params_nnexp}\n };\n \n std::string index_name = mxArrayToString(in_array[1]);\n if (index_table.find(index_name)==index_table.end()) {\n mexErrMsgTxt(\"Error: bad distance selector: wrong distance name.\");\n }\n FIndex* handle = mat2handle >(in_array[0]);\n (*index_table[index_name])(handle, in_n-2, in_array+2);\n //mxFree(index_name);\n}\n\ntemplate\nvoid _knn_search(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n FIndex* handle = mat2handle >(in_array[0]);\n int k = (int)(*mxGetPr(in_array[1]));\n //TODO: type check\n Matrix query = copy_matrix(in_array[2]);\n handle->knnSearch(k, query);\n}\n\ntemplate\nvoid _save_result(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n if ((in_n != 2) || (!mxIsChar(in_array[1]))) {\n mexErrMsgTxt(\"Error: bad path name\");\n }\n char* path = mxArrayToString(in_array[1]);\n FIndex* handle = mat2handle >(in_array[0]);\n handle->saveResults(path);\n mxFree(path);\n}\n\ntemplate\nstruct mexfunc {\n typedef void (*entry)(int, mxArray**, int, const mxArray**);\n};\n\nvoid mexFunction(int out_n, mxArray* out_array[], int in_n, const mxArray *in_array[]) {\n \n std::map::entry> ftable = {\n {\"new\", &_construct},\n {\"destruct\", &_destruct},\n {\"build_index\", &_build_index},\n {\"build_trees\", &_build_trees},\n {\"load_index\", &_load_index},\n {\"save_index\", &_save_index},\n {\"load_graph\", &_load_graph},\n {\"save_graph\", &_save_graph},\n {\"load_trees\", &_load_trees},\n {\"save_trees\", &_save_trees},\n {\"set_search_params\", &_set_search_params}, \n {\"knn_search\", &_knn_search}, \n {\"save_result\", &_save_result} \n// {\"set_distance_type\", &_set_distance_type} //TODO: only L2 distance is provided in general/distance.hpp now\n };\n\n if ((in_n<=1) || (!mxIsChar(in_array[0]))) {\n mexErrMsgTxt(\"Error: bad function selector: wrong type of function name.\");\n }\n std::string func_name = mxArrayToString(in_array[0]);\n if (ftable.find(func_name)==ftable.end()) {\n mexErrMsgTxt(\"Error: bad function selector: wrong function name.\");\n }\n mexPrintf(\"Running Function : %s ...\\n\", func_name.c_str());\n (*ftable[func_name])(out_n, out_array, in_n-1, in_array+1);\n //mxFree(func_name);\n}\n" }, { "path": "matlab/fvecs_read.m", "content": "% Read a set of vectors stored in the fvec format (int + n * float)\n% The function returns a set of output vector (one vector per column)\n%\n% Syntax: \n% v = fvecs_read (filename) -> read all vectors\n% v = fvecs_read (filename, n) -> read n vectors \n% v = fvecs_read (filename, [a b]) -> read the vectors from a to b (indices starts from 1)\nfunction v = fvecs_read (filename, bounds)\n\n% open the file and count the number of descriptors\nfid = fopen (filename, 'rb');\n \nif fid == -1\n error ('I/O error : Unable to open the file %s\\n', filename)\nend\n\n% Read the vector size\nd = fread (fid, 1, 'int');\nvecsizeof = 1 * 4 + d * 4;\n\n% Get the number of vectrors\nfseek (fid, 0, 1);\na = 1;\nbmax = ftell (fid) / vecsizeof;\nb = bmax;\n\nif nargin >= 2\n if length (bounds) == 1\n b = bounds;\n \n elseif length (bounds) == 2\n a = bounds(1);\n b = bounds(2); \n end\nend\n\nassert (a >= 1);\nif b > bmax\n b = bmax;\nend\n\nif b == 0 | b < a\n v = [];\n fclose (fid);\n return;\nend\n\n% compute the number of vectors that are really read and go in starting positions\nn = b - a + 1;\nfseek (fid, (a - 1) * vecsizeof, -1);\n\n% read n vectors\nv = fread (fid, (d + 1) * n, 'float=>single');\nv = reshape (v, d + 1, n);\n\n% Check if the first column (dimension of the vectors) is correct\nassert (sum (v (1, 2:end) == v(1, 1)) == n - 1);\nv = v (2:end, :);\n \nfclose (fid);\n" }, { "path": "matlab/handle_wrapper.hpp", "content": "//#define char16_t LIBRARY_char16_t\n#include \n//#undef char16_t\n\ntemplate class handle_wrapper\n{\npublic:\n handle_wrapper(base *ptr) : handle(ptr) {}\n ~handle_wrapper() {delete handle;}\n base *get_handle() {return handle;}\nprivate:\n base *handle;\n};\n\ntemplate inline mxArray *handle2mat(base *ptr)\n{\n mxArray *mat = mxCreateNumericMatrix(1, 1, mxUINT64_CLASS, mxREAL);\n *((uint64_t *)mxGetData(mat)) = reinterpret_cast(new handle_wrapper(ptr));\n return mat;\n}\n\ntemplate inline handle_wrapper *mat2wrapper(const mxArray *mat)\n{\n if (mxGetNumberOfElements(mat) != 1 || mxGetClassID(mat) != mxUINT64_CLASS || mxIsComplex(mat))\n mexErrMsgTxt(\"Input must be a real uint64 scalar.\");\n handle_wrapper *wrapper = reinterpret_cast *>(*((uint64_t *)mxGetData(mat)));\n return wrapper;\n}\n\ntemplate inline base *mat2handle(const mxArray *mat)\n{\n return mat2wrapper(mat)->get_handle();\n}\n\ntemplate inline void delete_wrapper(const mxArray *mat)\n{\n delete mat2wrapper(mat);\n}\n" }, { "path": "matlab/samples/example_buildall.m", "content": "% add the path of efanna library\naddpath('../')\n\n% use fvecs_read util function provided to load datasets\ndataset = fvecs_read('~/data/sift/sift_base.fvecs');\ndisp('Data size:');\ndisp(size(dataset));\n% params: dataset, number of trees for overall building, conquer-to-depth, number of iterations, L, checkK, K, S, number of trees for graph building\n% click following link for more information: https://github.com/fc731097343/efanna/blob/master/README.md\nef = efanna(dataset, 16, 8, 8, 30, 25, 10, 10, 8);\n% build graph and get a sparse matrix describing the NN results\nspmat = ef.build_index();\ndisp('Adjacency matrix of KNN graph acquired. Shape:');\ndisp(size(spmat));\ndisp('Number of non-zero elements in adjacency matrix of KNN graph:');\ndisp(nnz(spmat));\n% save trees and graph\nef.save_trees('./sift.trees');\nef.save_graph('./sift.graph');\n" }, { "path": "matlab/samples/example_buildgraph.m", "content": "% add the path of efanna library\naddpath('../')\n\n% use fvecs_read util function provided to load datasets\ndataset = fvecs_read('~/data/sift/sift_base.fvecs');\ndisp('Data size:');\ndisp(size(dataset));\n% params: dataset, number of trees for graph building, conquer-to-depth, number of iterations, L, checkK, K, S\n% click following link for more information: https://github.com/fc731097343/efanna/blob/master/README.md\nef = efanna(dataset, 8, 8, 7, 30, 25, 10, 10);\n% build graph and get a sparse matrix describing the NN results\n% the sparse matrix is row-wise organized, i.e. the first row lies the 0-1 vector describes k nearest neighbours for node 1.\ntic;\nspmat = ef.build_index();\ntoc;\n\ngdgraph = ivecs_read('~/data/sift/sift_10NN_groundtruth.graph');\nspmat = spmat';\n[row,col]=find(spmat);\nrow = row-1;\nnCorrect = 0;\nfor i=1:1000000\n for j=1:10\n if(find(gdgraph(:,i)==row((i-1)*10+j)))\n nCorrect = nCorrect + 1;\n end\n end\nend \ndisp(['10NN accuracy: ',num2str(nCorrect/10000000)]);\n% save graph\nef.save_graph('./sift.graph');\n" }, { "path": "matlab/samples/example_buildtree.m", "content": "% add the path of efanna library\naddpath('../')\n\n% use fvecs_read util function provided to load datasets\ndataset = fvecs_read('~/data/sift/sift_base.fvecs');\ndisp('Data size:');\ndisp(size(dataset));\n% params: dataset, number of trees to build\n% click following link for more information: https://github.com/fc731097343/efanna/blob/master/README.md\nef = efanna(dataset, 16);\n% build trees and save\nef.build_trees();\nef.save_trees('./sift.trees');\n" }, { "path": "matlab/samples/example_search.m", "content": "% add the path of efanna library\naddpath('../')\n\n% use fvecs_read util function provided to load datasets and queries\ndataset = fvecs_read('~/data/sift/sift_base.fvecs');\ndisp('Data size:');\ndisp(size(dataset));\nquery = fvecs_read('~/data/sift/sift_query.fvecs');\ndisp('Query size:');\ndisp(size(query));\n% params: dataset, number of trees for overall building, conquer-to-depth, number of iterations, L, checkK, K, S, number of trees for graph building\n% all params should be strictly corresponds to the params in building process of loaded graph and trees\n% click following link for more information: https://github.com/fc731097343/efanna/blob/master/README.md\nef = efanna(dataset, 16, 8, 8, 30, 25, 10, 10, 8);\n% build, or load graph & trees\nef.load_trees('./sift.trees');\nef.load_graph('./sift.graph');\n%%% ef.build_index();\n\n% search params: number of trees to use, number of epoches, pool size factor, extend factor, searching methods\n% more details can be found in README.md of efanna root as well.\nef.set_search_params(16, 4, 1200, 200, 0);\n% params: required number of returned neighbors (i.e. k for knn, here are searching for 10-nn)\nef.knn_search(10, query);\nef.save_result('./sift_search.result');\n" }, { "path": "samples/efanna_index_buildall.cc", "content": "#include \n#include \n#include \n#include \n#include \n\nusing namespace efanna;\nusing namespace std;\nvoid load_data(char* filename, float*& data, size_t& num,int& dim){// load data with sift10K pattern\n ifstream in(filename, ios::binary);\n if(!in.is_open()){cout<<\"open file error\"< dataset(points_num,dim,data_load);\n //Matrix query(q_num,qdim,query_load);\n\n unsigned int trees = atoi(argv[4]);\n int mlevel = atoi(argv[5]);\n unsigned int epochs = atoi(argv[6]);\n int L = atoi(argv[7]);\n int checkK = atoi(argv[8]);\n int kNN = atoi(argv[9]);\n int S = atoi(argv[10]);\n\n //srand(time(NULL));\n FIndex index(dataset, new L2DistanceAVX(), efanna::KDTreeUbIndexParams(true, trees ,mlevel ,epochs,checkK,L, kNN, trees, S));\n clock_t s,f;\n s = clock();\n index.buildIndex();\n\n f = clock();\n cout<<\"Index building time : \"<<(f-s)*1.0/CLOCKS_PER_SEC<<\" seconds\"<\n#include \n#include \n#include \n#include \n#include \n\nusing namespace efanna;\nusing namespace std;\nvoid load_data(char* filename, float*& data, size_t& num,int& dim){// load data with sift10K pattern\n ifstream in(filename, ios::binary);\n if(!in.is_open()){cout<<\"open file error\"< dataset(points_num,dim,data_load);\n //Matrix query(q_num,qdim,query_load);\n\n unsigned int trees = atoi(argv[3]);\n int mlevel = atoi(argv[4]);\n unsigned int epochs = atoi(argv[5]);\n int L = atoi(argv[6]);\n int checkK = atoi(argv[7]);\n int kNN = atoi(argv[8]);\n int S = atoi(argv[9]);\n\n //srand(time(NULL));\n FIndex index(dataset, new L2DistanceAVX(), efanna::KDTreeUbIndexParams(true, trees ,mlevel ,epochs,checkK,L, kNN, trees, S));\nboost::timer::auto_cpu_timer timer;\n\n index.buildIndex();\ncout<\n#include \n#include \n#include \n#include \n\nusing namespace efanna;\nusing namespace std;\nvoid load_data(char* filename, float*& data, size_t& num,int& dim){// load data with sift10K pattern\n ifstream in(filename, ios::binary);\n if(!in.is_open()){cout<<\"open file error\"< dataset(points_num,dim,data_load);\n\n unsigned int trees = atoi(argv[3]);\n\n FIndex index(dataset, new L2DistanceAVX(), efanna::KDTreeUbIndexParams(true, trees ,8 ,8,25,30, 10, 8, 10));\n clock_t s,f;\n s = clock();\n index.buildTrees();\n\n f = clock();\n cout<<\"Index building time : \"<<(f-s)*1.0/CLOCKS_PER_SEC<<\" seconds\"<\n#include \n#include \n#include \n#include \n#include \n\nusing namespace boost;\nusing namespace efanna;\nusing namespace std;\nvoid load_data(char* filename, float*& data, size_t& num,int& dim){// load data with sift10K pattern\n ifstream in(filename, ios::binary);\n if(!in.is_open()){cout<<\"open file error\"< dataset(points_num,dim,data_load);\n Matrix query(q_num,qdim,query_load);\n\n FIndex index(dataset, new L2DistanceAVX(), efanna::KDTreeUbIndexParams(true, 8 ,8 ,10,25,30,10));\n index.loadTrees(argv[2]);\n index.loadGraph(argv[3]);\n\n int search_trees = atoi(argv[6]);\n int search_epoc = atoi(argv[7]);\n int poolsz = atoi(argv[8]);\n int search_extend = atoi(argv[9]);\n int search_method = argc == 12 ? atoi(argv[11]) : 0;\n index.setSearchParams(search_epoc, poolsz, search_extend, search_trees, search_method);\n\n\n//clock_t s,f;\nboost::timer::auto_cpu_timer timer;\n//s=clock();\n index.knnSearch(atoi(argv[10])/*query nn*/,query);\n//f=clock();\n\n//cout<<(f-s)*1.0/CLOCKS_PER_SEC/8<\n#include \n#include \n\nusing namespace std;\nvoid load_data(char* filename, int*& data, int& num,int& dim){// load data with sift10K pattern\n ifstream in(filename, ios::binary);\n if(!in.is_open()){cout<<\"open file error\"< dim1){cout<< \"result file not enough for k=\"< result;\n for(int i=0; i < p1; i++){\n result.clear();\n for(int j=0; j